LatentLoop reallocates compute across looped MoE passes
LatentLoop compresses the sequence during a looped Mixture-of-Experts model's first pass, keeps the second at full resolution, and uses the saved compute to increase the hidden size. It shares MoE weights across passes while using separate attention parameters, and the authors report lower validation NLL across 15 configurations from 1.3B to 7B parameters, along with improved average scores on nine benchmarks.
The paper presents asymmetric compute allocation as a way to improve looped MoE models, with predicted compute savings of 19.0% versus vanilla MoE and 7.9% versus a looped baseline at a fitted NLL of 1.988.
Computer Science > Machine Learning
arXiv:2610.19975 (cs)
[Submitted on 14 Oct 2026]
Title:LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation
Authors:Zefeng He, Yunkai Chai, Xiaoyang Chen, Tong Zhu, Zongliang Wu, Yun Luo, Yafu Li, Yu Cheng
View a PDF of the paper titled LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation, by Zefeng He and 7 other authors