Hugging Face Trainer

Headlines · 1

1. Nov 2
   Near-Zero Embeddings Reportedly Stall Gated-Model Training

   A paper by Xin Liu and five coauthors surveys 291 open-weight language-model repositories, finding that 52.6% of untied gated and hybrid checkpoints contain reachable input embeddings with norms below 10^-6, compared with 0% of weight-tied models. The authors report that, in gated sequence models such as Mamba, Gated DeltaNet and xLSTM, near-zero embeddings can amplify input gradients, cause FP32 clipping to overflow to infinity and discard a batch’s gradients. During Telugu continual pretraining of Nemotron-3-Nano-4B, the paper reports discarded updates on approximately 20% of steps. The authors propose GroupClip for released checkpoints and GateFloor for pretraining, reporting recovered updates, reduced near-zero gradients and comparable perplexity in their experiments.

   Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my Hugging Face Trainer experienceAsk about Hugging Face Trainer