Mixture-of-Experts

Headlines · 2

1. 3d
   AIR schedule targets effective learning rate in weight-normalized training

   An arXiv preprint proposes AIR (Affine Inverse-square-Root decay), a learning-rate schedule that targets the effective learning rate rather than the nominal learning rate alone. AIR combines a theoretically derived inverse-square-root decay with an affine adjustment to meet prescribed peak and terminal learning rates. The authors report that, in 128B-token Mixture-of-Experts pretraining, AIR outperformed cosine and other candidate schedules in training loss, downstream performance, and hyperparameter robustness, with terminal training loss approximately 0.01 lower than cosine at matched cumulative learning rate.

   Hugging Face · Papers · 🔥 0
2. 5d
   Calibration-Free Top-k Selection Reduces MoE Expert Reads

   A paper by Geunhyoung Kim and Sung Ju Hwang proposes top-k channel selection for Mixture-of-Experts decoding, avoiding the offline calibration data used by threshold-based selectors. The authors report that top-k matched or exceeded threshold variants in 40 of 43 comparisons at 13% and 18% read budgets, including all 23 comparisons on generation tasks, and avoided repetition-loop collapse observed with calibrated thresholds on Gemma 4. Their separate reader retained 98.1% of full-model accuracy while reading 17.8% of routed-expert weights across three post-trained MoE models. The authors also report that its Triton kernels ran MoE blocks 1.60–1.90 times faster than optimized vLLM at batch size one and up to 2.69 times faster at larger batch sizes.

   Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my Mixture-of-Experts experienceAsk about Mixture-of-Experts