T-Tech展示LIMe降低大模型训练算力
AI 总结
T-Tech研究人员在EMNLP 2026介绍了LIMe，称其是一种轻量级注意力扩展，通过复用各层的键值缓存，并为每个注意力头和层学习路由权重。在作者约10亿参数模型的实验中，LIMe在使用约少15%算力的情况下达到基线LLaMA相同的训练损失；作者还报告其在语言建模、多步推理和合成算术基准上的表现提升。报道附有论文和官方实现，但这些结果主要来自作者实验。
深度见解
LIMe提出了一种在保持隐藏状态大小不变的情况下减少语言模型训练算力的注意力方案，作者称其在更深架构和合成算术任务上的改进尤其明显。
| At EMNLP 2026, an A* conference currently taking place in Hungary, researchers from T-Tech presented LIMe, a method that makes it possible to train language models with approximately 15% less compute without compromising quality. In traditional Transformers, each layer relies solely on the hidden state of the previous layer, meaning all contextual information must pass through it. The authors demonstrate that this creates pressure toward representation collapse: tokens become increasingly similar to one another, their distinguishing features fade, and model quality suffers as a result. LIMe (Layer-Integrated Memory) is a lightweight extension of the attention mechanism. It leverages the key-value buffers already maintained for each layer and introduces learnable routing weights specific to each attention head and layer. These weights allow attention heads to retrieve and combine representations not only from the immediately preceding layer but also from earlier layers. The hidden state size remains unchanged, while previously computed information is reused. In experiments with models containing approximately one billion parameters, LIMe achieved the same training loss as baseline LLaMA while using 15% less compute. The method also outperformed competing approaches on language modeling and multi-step reasoning benchmarks. The most significant improvements were observed in synthetic arithmetic tasks and architectures with greater depth. The authors also report that LIMe produces higher entropy in value vectors and better token separability, suggesting that it effectively mitigates the underlying representation collapse problem. Analysis of the learned routing weights further reveals that the model systematically reuses both local and long-range features. Paper: https://arxiv.org/abs/2502.09245 Official implementation: https://github.com/corl-team/lime submitted by /u/ChampionshipLimp1749 [link] [comments] |