创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

Reddit实测未发现明显位置敏感现象

AI 总结

一名 Reddit 用户通过把含答案的文档放在不同位置,测试 Haiku 4.5 和 Sonnet 5 的回答是否会随位置改变。测试使用5、15和30篇短文档;两种模型在5篇和15篇时均未出现答案变化,30篇时约4%的表面变化仅是“3”与“three”的格式差异。作者强调,该结果只覆盖简单单事实检索,未测试长文档、多跳问题或证据不明显的任务。

深度见解

这项实验为当前 Claude 模型在简单检索中的位置稳定性提供了可复现但范围有限的信号,不能据此认定更广泛的“中间迷失”问题已经解决。

Everyone keeps citing lost in the middle so, instead of going by the paper's citations, i wanted to actually check it on my own stuff, without building a labeled set for it.

So i did the lazy version. make up facts the model can’t already know, drop the one document that has the answer into a pile of distractor documents, and slide that doc through every position (front, middle, back), asking the question each time. if the answer changes depending on where the doc sits, that’s position sensitivity. no ground truth needed, just checking it doesn’t contradict itself.

ran it on haiku 4.5 and sonnet 5 at 5, 15 and 30 docs.

Found actually nothing. 0 flips at 5 and 15 on both models. at 30 docs i got about 4%, then actually looked at what “flipped” and it was the model saying “3” vs “three” for the same question!! same correct answer, just a different format. once i stopped counting that it went back to 0.

so at least on easy single-fact retrieval with short docs, current models seem fine with position up to 30 docs. couldn’t reproduce the scary version.

caveats, obviously this is the easy case. one clear fact, short docs. i’d bet it still shows up with long documents or multi-hop or when the relevant bit is less obvious, i just didn’t test that. also if you try this yourself don’t use exact string match, you’ll get fooled by 3 vs three like i did. compare meaning not strings.

script’s on github if you want to run it on your own setup: github.com/tarunagarwal1981/wobbly (scripts/live_rag.py)

has anyone actually reproduced lost in the middle on a current model? what context length or task did it show up at for you? genuinely curious if it’s still a thing or mostly solved on the easy cases now.

#Haiku 4.5#Sonnet 5#wobbly

原文约 1 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。