复测称新版Claude双方NTA率升高
一名Reddit发帖者称，其朋友重新运行了斯坦福的AITA基准测试，共测试27个模型，并将Reddit判定为NTA的帖子改写为另一方视角。新版Claude中，Sonnet 5双方NTA率为35%，Sonnet 5.5为30%，较早的Opus 4.1为14%；Grok 4.20为73%，Gemini 4 Argon为7%。发帖者关于Claude已成为同类模型中最差的概括强于文中数据，目前结果仍属二手陈述，尚待独立核验。
双方NTA率可直接衡量模型迎合用户表述的程度，因此这次复测结果与模型迎合性及行为评估密切相关。
One of my friends re-ran a Stanford study on 27 models. In the study, they ask AI, "Am I the asshole?" from both sides of the same fight. I found the read interesting and wanted to share it with you too.
So, they take an AITA post where Reddit said NTA, rewrite it from the other person's side, and ask the model about both. If it says NTA to both people, it's just agreeing with whoever is asking.
Opus 4.1 only did this 14% of the time, the best of any model back then. Most other labs have improved a lot since then, but Claude went the other way. Sonnet 5 is at 35%, and Sonnet 5.5 is the highest of the newest models with 30%. I always thought of Claude as the one that pushes back; now I'll probably discuss my relationships with it.
Grok 4.20 was the worst of all 27 at 73%, worse than GPT-3.5 (from 2023). Grok 4.7 came out six months later, and it's at 20%. I will attribute this speed to testing it on Elon Musk, but it's just my speculation.
Finally, Gemini 4 Argon has the lowest score (7%), but partly because it refused to pick a side on 31 of the stories. Every refusal was on the other person's version, never the original poster's
One example from the posts: a 4-year-old won't hug his dad goodnight, so the dad sits on the floor with his head in his hands, making sad noises. All 27 models said the mom was right to call it out. 8 of them also told the dad he was NTA
You can find the full results; you can replicate them yourself here: heldoutlabs.com/ai-am-i-the-asshole-test
submitted by /u/ahmetfirat [link] [comments]
#Claude Opus 4.1#Claude Sonnet 5#Claude Sonnet 5.5#Grok 4.20#Grok 4.7#Gemini 4 Argon#GPT-3.5#Stanford University