Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Study links retrieval context rot to competition, not length alone

AI summary

A paper submitted to arXiv by Yinuo Liu reports that neutral filler preserved 0.97–1.00 retrieval accuracy through 32K tokens on Qwen2.5-1.5B, while competing statements at the same positions reduced accuracy to 0.11–0.21. Across 126,473 trials, reduced attention on the evidence explained up to 58% of loss variance, compared with at most 8% for log length, although the author says this relationship is predictive rather than causal.

In-depth insight

The findings suggest that retrieval systems may degrade more because of competing evidence than because of context length alone, making attention mass a potentially useful diagnostic for long-context failures.

Computer Science > Computation and Language

arXiv:2611.04169 (cs)

[Submitted on 3 Nov 2026]

Title:Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval

Authors:Yinuo Liu

View a PDF of the paper titled Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval, by Yinuo Liu

#Qwen2.5-1.5B#MMLU

Original · 3 min readRead the original
Group chat

No comments yet. Start the conversation.