AIR

Headlines · 1

1. 3d
   AIR schedule targets effective learning rate in weight-normalized training

   An arXiv preprint proposes AIR (Affine Inverse-square-Root decay), a learning-rate schedule that targets the effective learning rate rather than the nominal learning rate alone. AIR combines a theoretically derived inverse-square-root decay with an affine adjustment to meet prescribed peak and terminal learning rates. The authors report that, in 128B-token Mixture-of-Experts pretraining, AIR outperformed cosine and other candidate schedules in training loss, downstream performance, and hyperparameter robustness, with terminal training loss approximately 0.01 lower than cosine at matched cumulative learning rate.

   Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my AIR experienceAsk about AIR