Reddit test finds no clear position sensitivity in Haiku 4.5 or Sonnet 5
A Reddit user tested whether moving the document containing an answer changes a model’s response by placing it among distractors at different positions. The experiment used Haiku 4.5 and Sonnet 5 with 5, 15 and 30 short documents; both models recorded zero answer flips at 5 and 15 documents, while the apparent 4% flips at 30 documents were only variations between “3” and “three.” The author says the result applies to easy, single-fact retrieval and did not test long documents, multi-hop questions or less obvious evidence.
The experiment offers a reproducible but narrow signal that current Claude models may resist document-position effects in simple retrieval, while not establishing that the broader “lost in the middle” problem has been solved.
Everyone keeps citing lost in the middle so, instead of going by the paper's citations, i wanted to actually check it on my own stuff, without building a labeled set for it.
So i did the lazy version. make up facts the model can’t already know, drop the one document that has the answer into a pile of distractor documents, and slide that doc through every position (front, middle, back), asking the question each time. if the answer changes depending on where the doc sits, that’s position sensitivity. no ground truth needed, just checking it doesn’t contradict itself.
ran it on haiku 4.5 and sonnet 5 at 5, 15 and 30 docs.
Found actually nothing. 0 flips at 5 and 15 on both models. at 30 docs i got about 4%, then actually looked at what “flipped” and it was the model saying “3” vs “three” for the same question!! same correct answer, just a different format. once i stopped counting that it went back to 0.
so at least on easy single-fact retrieval with short docs, current models seem fine with position up to 30 docs. couldn’t reproduce the scary version.
caveats, obviously this is the easy case. one clear fact, short docs. i’d bet it still shows up with long documents or multi-hop or when the relevant bit is less obvious, i just didn’t test that. also if you try this yourself don’t use exact string match, you’ll get fooled by 3 vs three like i did. compare meaning not strings.
script’s on github if you want to run it on your own setup: github.com/tarunagarwal1981/wobbly (scripts/live_rag.py)
has anyone actually reproduced lost in the middle on a current model? what context length or task did it show up at for you? genuinely curious if it’s still a thing or mostly solved on the easy cases now.