Latent Feedback May Improve Language-Model Training Efficiency
An arXiv preprint reports that latent feedback, which feeds each token’s top-layer state into the next token’s input, can help language models reach a standard Transformer’s loss with less compute or unique data in tested settings. The authors report maximum savings of 48% in compute and 47% in unique data, with savings increasing with unique data per parameter and repetition; they also say the benefit is reduced when unique data is scarce relative to model size. A scaling law fitted on models from 60 million to 271 million parameters predicted losses within 0.3% and model gaps within 0.05% at a held-out 778-million-parameter size.
Computer Science > Machine Learning
arXiv:2611.20404 (cs)
[Submitted on 16 Nov 2026]
Title:Latent Feedback Is a Compute and Data Amplifier
Authors:Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu
View a PDF of the paper titled Latent Feedback Is a Compute and Data Amplifier, by Benhao Huang and 4 other authors