Coding-Agent Harness Changes Raise Costs Without Reliable Gains
Shandong University researchers Liu and Han compared Claude Code, OpenCode, and mini-SWE-agent while holding the language model constant. Most performance differences were similar to rerun noise, while the harnesses consumed two to three times as many tokens as the leanest option; OpenCode showed a significant deficit on the larger task pool. A small cross-check on Terminal-Bench 2.1 showed the same pattern.
The study suggests that coding-agent leaderboards should publish repeated runs, because single-run gaps may reflect benchmark noise rather than better harness design.
Fancy harnesses make coding agents more expensive but not better, study finds
Jonathan Kemper View the LinkedIn Profile of Jonathan Kemper
Oct 24, 2026
Nano Banana Pro prompted by THE DECODER
Makers tout score gains from new agent harnesses like Claude Code or OpenCode. A study finds that claim doesn't hold up. Swapping harnesses barely moves results more than running the same setup twice, but costs can triple.
#Shandong University#Claude Code#OpenCode#mini-SWE-agent#Claude Opus 5#Qwen3.6#Qwen3.8#Deepseek-V4-Flash#GLM-5.3-Flash#HY4-Preview#SWE-bench Verified#Terminal-Bench 2.1