Taste-Bench
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- Taste-Bench: AI agents pick the better long-horizon path only ~60% of the time
Researchers from Microsoft and City University of Hong Kong built Taste-Bench, which tests an agent's "taste" in long-running tasks by freezing a run at a "decision fork," hiding later work and asking a model to choose between two diverging paths. The benchmark has 502 binary questions drawn from thousands of scored agent runs on coding and AI research tasks, including SWE-bench Pro and METR's MALT transcripts. Across 14 models, GPT-5.6 Sol led with 59.7% and GPT-5.5 followed at 59.5%, while average accuracy fell to 21%, near random, when the deciding clue appeared only after extensive later work. The team also distilled a teacher's judgment into a Qwen3.6-27B student, lifting accuracy on unseen tasks from 30% to 47.9% and raising success on 41 held-out SWE-bench Pro tasks from 14.6% to 33.7% when its advice was added to the task context. The benchmark and code are publicly available.
The Decoder · 🔥 0
Experience and discussion from the community
Share my Taste-Bench experienceAsk about Taste-Bench
Nobody has shared their experience with Taste-Bench yet.