Reddit实测未发现明显位置敏感现象
一名 Reddit 用户通过把含答案的文档放在不同位置,测试 Haiku 4.5 和 Sonnet 5 的回答是否会随位置改变。测试使用5、15和30篇短文档;两种模型在5篇和15篇时均未出现答案变化,30篇时约4%的表面变化仅是“3”与“three”的格式差异。作者强调,该结果只覆盖简单单事实检索,未测试长文档、多跳问题或证据不明显的任务。
这项实验为当前 Claude 模型在简单检索中的位置稳定性提供了可复现但范围有限的信号,不能据此认定更广泛的“中间迷失”问题已经解决。
Everyone keeps citing lost in the middle so, instead of going by the paper's citations, i wanted to actually check it on my own stuff, without building a labeled set for it.
So i did the lazy version. make up facts the model can’t already know, drop the one document that has the answer into a pile of distractor documents, and slide that doc through every position (front, middle, back), asking the question each time. if the answer changes depending on where the doc sits, that’s position sensitivity. no ground truth needed, just checking it doesn’t contradict itself.
ran it on haiku 4.5 and sonnet 5 at 5, 15 and 30 docs.
Found actually nothing. 0 flips at 5 and 15 on both models. at 30 docs i got about 4%, then actually looked at what “flipped” and it was the model saying “3” vs “three” for the same question!! same correct answer, just a different format. once i stopped counting that it went back to 0.
so at least on easy single-fact retrieval with short docs, current models seem fine with position up to 30 docs. couldn’t reproduce the scary version.
caveats, obviously this is the easy case. one clear fact, short docs. i’d bet it still shows up with long documents or multi-hop or when the relevant bit is less obvious, i just didn’t test that. also if you try this yourself don’t use exact string match, you’ll get fooled by 3 vs three like i did. compare meaning not strings.
script’s on github if you want to run it on your own setup: github.com/tarunagarwal1981/wobbly (scripts/live_rag.py)
has anyone actually reproduced lost in the middle on a current model? what context length or task did it show up at for you? genuinely curious if it’s still a thing or mostly solved on the easy cases now.