T-Tech Presents LIMe for Lower LLM Training Compute
T-Tech researchers presented LIMe at EMNLP 2026, describing it as a lightweight attention extension that reuses key-value buffers and learns routing weights for each attention head and layer. In author-reported experiments with approximately one-billion-parameter models, LIMe matched baseline LLaMA training loss while using about 15% less compute, and the authors reported stronger results on language-modeling, multi-step-reasoning and synthetic-arithmetic benchmarks. The report also links the paper and an official implementation, but the findings primarily come from the authors’ experiments.
LIMe offers a concrete attention-based approach to reducing language-model training compute without changing hidden-state size, with reported gains especially pronounced in deeper architectures and synthetic arithmetic tasks.
| At EMNLP 2026, an A* conference currently taking place in Hungary, researchers from T-Tech presented LIMe, a method that makes it possible to train language models with approximately 15% less compute without compromising quality. In traditional Transformers, each layer relies solely on the hidden state of the previous layer, meaning all contextual information must pass through it. The authors demonstrate that this creates pressure toward representation collapse: tokens become increasingly similar to one another, their distinguishing features fade, and model quality suffers as a result. LIMe (Layer-Integrated Memory) is a lightweight extension of the attention mechanism. It leverages the key-value buffers already maintained for each layer and introduces learnable routing weights specific to each attention head and layer. These weights allow attention heads to retrieve and combine representations not only from the immediately preceding layer but also from earlier layers. The hidden state size remains unchanged, while previously computed information is reused. In experiments with models containing approximately one billion parameters, LIMe achieved the same training loss as baseline LLaMA while using 15% less compute. The method also outperformed competing approaches on language modeling and multi-step reasoning benchmarks. The most significant improvements were observed in synthetic arithmetic tasks and architectures with greater depth. The authors also report that LIMe produces higher entropy in value vectors and better token separability, suggesting that it effectively mitigates the underlying representation collapse problem. Analysis of the learned routing weights further reveals that the model systematically reuses both local and long-range features. Paper: https://arxiv.org/abs/2502.09245 Official implementation: https://github.com/corl-team/lime submitted by /u/ChampionshipLimp1749 [link] [comments] |