Ai2发布XYBench评估助手处理错误假设
Ai2与Carnegie Mellon、NVIDIA及Johns Hopkins研究人员发布XYBench,包含8,115个涉及编程、烹饪、家居维修等实际任务的题目。该基准评估助手能否识别误区、处理用户的潜在目标,并优先提供有效方案。Ai2称,Claude Sonnet 4.6、GPT-5、Gemini 3.1 Pro、Qwen3.5-9B和DeepSeek-V4 Pro经常沿用用户的错误思路,同时该研究获得了EMNLP 2026最佳资源论文奖。
XYBench为衡量AI助手能否识别错误假设、围绕用户真正目标提供建议,以及突破仅回答字面问题的表现,提供了具体评测方法。
XYBench: Helping with the problem behind the question
November 13, 2026
Ai2
Truly useful AI assistants need to be able to help users who ask misguided questions—not just users who ask the right question with the right amount of detail. We’re releasing XYBench, a benchmark of 8K+ questions that tests how AI assistants respond to requests based on misconceptions, such as asking how to thicken frosting that keeps sliding off cupcakes when the real problem is that the cupcakes are still too warm to frost. XYBench won the Best Resource Paper Award at EMNLP 2026.
#XYBench#Ai2#Carnegie Mellon#NVIDIA#Johns Hopkins#Claude Sonnet 4.6#GPT-5#Gemini 3.1 Pro#Qwen3.5-9B#DeepSeek-V4 Pro