Near-Zero Embeddings Reportedly Stall Gated-Model Training
A paper by Xin Liu and five coauthors surveys 291 open-weight language-model repositories, finding that 52.6% of untied gated and hybrid checkpoints contain reachable input embeddings with norms below 10^-6, compared with 0% of weight-tied models. The authors report that, in gated sequence models such as Mamba, Gated DeltaNet and xLSTM, near-zero embeddings can amplify input gradients, cause FP32 clipping to overflow to infinity and discard a batch’s gradients. During Telugu continual pretraining of Nemotron-3-Nano-4B, the paper reports discarded updates on approximately 20% of steps. The authors propose GroupClip for released checkpoints and GateFloor for pretraining, reporting recovered updates, reduced near-zero gradients and comparable perplexity in their experiments.
The paper identifies a concrete mechanism by which near-zero embeddings can interrupt updates in gated sequence models and reports two remedies for different stages of the model lifecycle.
Computer Science > Machine Learning
arXiv:2611.04449 (cs)
[Submitted on 3 Nov 2026]
Title:Near-Zero Token Embeddings Silently Stall Training in Gated Sequence Models
Authors:Xin Liu, Boyuan Sun, Yuxin Wang, Haoran Song, Mingjian Zhao, Mengyu Liu
View a PDF of the paper titled Near-Zero Token Embeddings Silently Stall Training in Gated Sequence Models, by Xin Liu and 5 other authors
#Mamba#Gated DeltaNet#xLSTM#Nemotron-3-Nano-4B#GroupClip#GateFloor#PyTorch#Hugging Face Trainer#FSDP2#AdamW