Ai2 releases XYBench for testing assistants on mistaken assumptions
Ai2, with researchers from Carnegie Mellon, NVIDIA, and Johns Hopkins, released XYBench, a benchmark containing 8,115 questions about programming, cooking, home repair, and other practical tasks. It evaluates whether assistants identify misconceptions, address the user’s underlying goal, and prioritize effective solutions over the requested approach. Ai2 reports that Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro often retained users’ mistaken approaches and that XYBench received the Best Resource Paper Award at EMNLP 2026.
XYBench gives researchers a specific way to measure whether AI assistants can recognize flawed assumptions and provide advice aimed at the user’s actual goal, rather than merely answering the literal request.
XYBench: Helping with the problem behind the question
November 13, 2026
Ai2
Truly useful AI assistants need to be able to help users who ask misguided questions—not just users who ask the right question with the right amount of detail. We’re releasing XYBench, a benchmark of 8K+ questions that tests how AI assistants respond to requests based on misconceptions, such as asking how to thicken frosting that keeps sliding off cupcakes when the real problem is that the cupcakes are still too warm to frost. XYBench won the Best Resource Paper Award at EMNLP 2026.
#XYBench#Ai2#Carnegie Mellon#NVIDIA#Johns Hopkins#Claude Sonnet 4.6#GPT-5#Gemini 3.1 Pro#Qwen3.5-9B#DeepSeek-V4 Pro