Taste-Bench: AI agents pick the better long-horizon path only ~60% of the time
Researchers from Microsoft and City University of Hong Kong built Taste-Bench, which tests an agent's "taste" in long-running tasks by freezing a run at a "decision fork," hiding later work and asking a model to choose between two diverging paths. The benchmark has 502 binary questions drawn from thousands of scored agent runs on coding and AI research tasks, including SWE-bench Pro and METR's MALT transcripts. Across 14 models, GPT-5.6 Sol led with 59.7% and GPT-5.5 followed at 59.5%, while average accuracy fell to 21%, near random, when the deciding clue appeared only after extensive later work. The team also distilled a teacher's judgment into a Qwen3.6-27B student, lifting accuracy on unseen tasks from 30% to 47.9% and raising success on 41 held-out SWE-bench Pro tasks from 14.6% to 33.7% when its advice was added to the task context. The benchmark and code are publicly available.
The authors say existing benchmarks only measure whether an agent finishes a task, not the quality of its steps along the way, and their results show accuracy near random when the evidence needed for a decision only appears later, with more reasoning budget not helping.
AI agents have bad taste in long-term decisions and a new benchmark proves it
Jonathan Kemper View the LinkedIn Profile of Jonathan Kemper
Oct 18, 2026
Nano Banana Pro prompted by THE DECODER
Researchers from Microsoft and City University of Hong Kong built a method to test how well AI agents make decisions during long-running tasks. The best model barely hit 60 percent.
#Microsoft#City University of Hong Kong#Taste-Bench#GPT-5.6 Sol#GPT-5.5#Qwen3.6-27B#SWE-bench Pro#METR