Muir linear-attention model targets long-context recall and streaming
The Muir paper introduces a linear-attention architecture that combines sensitivity-allocated gating, matched-filter readout, and a numerically stable parallel update. Its authors report lower perplexity and stronger recall than Gated DeltaNet-2 and Transformer baselines, along with 10M-token streaming from a 4K training length and near-perfect passkey retrieval up to 1M tokens. A Reddit analysis also describes a 25 MB state at 10M tokens, but these technical and benchmark claims have not been independently confirmed.
The paper targets a central limitation of recurrent and linear-attention models: retaining useful information over very long contexts without letting slow-decaying state rows or unstable training kernels dominate behavior.
Computer Science > Machine Learning
arXiv:2611.08376 (cs)
[Submitted on 5 Nov 2026]
Title:Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout
Authors:Tushaar Gangavarapu, Gil Tabak, Clemens JS Schaefer
View a PDF of the paper titled Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout, by Tushaar Gangavarapu and 2 other authors