AdamW

Headlines · 2

1. Nov 2
   Near-Zero Embeddings Reportedly Stall Gated-Model Training

   A paper by Xin Liu and five coauthors surveys 291 open-weight language-model repositories, finding that 52.6% of untied gated and hybrid checkpoints contain reachable input embeddings with norms below 10^-6, compared with 0% of weight-tied models. The authors report that, in gated sequence models such as Mamba, Gated DeltaNet and xLSTM, near-zero embeddings can amplify input gradients, cause FP32 clipping to overflow to infinity and discard a batch’s gradients. During Telugu continual pretraining of Nemotron-3-Nano-4B, the paper reports discarded updates on approximately 20% of steps. The authors propose GroupClip for released checkpoints and GateFloor for pretraining, reporting recovered updates, reduced near-zero gradients and comparable perplexity in their experiments.

   Hugging Face · Papers · 🔥 0
2. Oct 27
   ApexW PyTorch optimizer targets large GPU workloads

   A developer shared ApexW, a PyTorch optimizer that uses compact optimizer states and a Triton-based execution path for large GPU workloads. In the developer's optimizer-only tests, the Triton compact-state path was about 1.5 times faster than PyTorch's fused AdamW on workloads with a few large tensors while using roughly half the optimizer-state memory. The developer said these results did not represent full model training, that automatic mode did not consistently achieve the same speedup, and that workloads with many small tensors were much slower than PyTorch's fused optimizer. The developer also reported that PyTorch was slightly faster overall in an end-to-end MLP test.

   r/MachineLearning · 🔥 0

Experience and discussion from the community

Share my AdamW experienceAsk about AdamW