Muir以线性注意力改善长上下文召回
AI 总结
Muir论文提出一种结合敏感度分配门控、匹配滤波读出和数值稳定并行更新的线性注意力架构。作者称,该模型在困惑度和召回测试中优于Gated DeltaNet-2与Transformer,并能从4K训练长度外推处理1000万token,在100万token内保持接近满分的口令检索。Reddit分析还称其在1000万token下使用25 MB状态,但相关技术和基准结果尚未获独立确认。
深度见解
该研究聚焦线性注意力模型在超长上下文中的状态干扰与训练稳定性,试图在固定大小状态下保留更多有效信息。
Computer Science > Machine Learning
arXiv:2611.08376 (cs)
[Submitted on 5 Nov 2026]
Title:Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout
Authors:Tushaar Gangavarapu, Gil Tabak, Clemens JS Schaefer
View a PDF of the paper titled Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout, by Tushaar Gangavarapu and 2 other authors