Study links retrieval context rot to competition, not length alone
A paper submitted to arXiv by Yinuo Liu reports that neutral filler preserved 0.97–1.00 retrieval accuracy through 32K tokens on Qwen2.5-1.5B, while competing statements at the same positions reduced accuracy to 0.11–0.21. Across 126,473 trials, reduced attention on the evidence explained up to 58% of loss variance, compared with at most 8% for log length, although the author says this relationship is predictive rather than causal.
The findings suggest that retrieval systems may degrade more because of competing evidence than because of context length alone, making attention mass a potentially useful diagnostic for long-context failures.
Computer Science > Computation and Language
arXiv:2611.04169 (cs)
[Submitted on 3 Nov 2026]
Title:Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval
Authors:Yinuo Liu
View a PDF of the paper titled Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval, by Yinuo Liu