Co-Design Bench exposes gaps in real-world coding agents
Alibaba researchers and researchers from several Asian universities built Co-Design Bench from real coding-change requests, including multimodal inputs such as images, audio, video, and documents. The paper reports that GPT-6 Astra led the ten tested models by satisfying about 70% of task requirements on average, but was fully correct in only 26.4% of tasks; performance also varied with the agent harness.
The benchmark provides a stricter test for coding agents on realistic web-app changes, showing that leading models still leave work for manual cleanup while making the operating environment a material part of evaluation.
AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test
Jonathan Kemper View the LinkedIn Profile of Jonathan Kemper
Nov 8, 2026
ChatGPT-5.6 prompted by THE DECODER
Researchers from Alibaba and several Asian universities built a coding-agent benchmark from real user change requests. Even the best model fully completes only about one in four tasks, putting the near-perfect scores of older tests into perspective.
#Co-Design Bench#Alibaba#GPT-6 Astra#Claude Opus 5#Qwen3.8-Max#Qwen3.8-27B#Grok 4.6#GLM-5.2#WebCompass-edit#SWE-bench Verified