Opus-led coding teams did not improve cost-performance
In a Reddit experiment by author /u/Fabulous_Pollution10, Claude Opus 5.5 alone solved 81% of 12 hard coding tasks at $1.20 per task, while pairing it with DeepSeek V4.1 Flash produced 67% at $1.29. Across seven team designs, 41 setups and roughly 1,200 runs, none beat Opus alone and most cost more; the author attributes some failures to weak workers, handoff errors and prompt-cache rebuilding.
The reported results suggest that simply assigning planning and review to an expensive model while delegating implementation to cheaper models may not improve AI coding economics, although the small task sample limits confidence in small performance differences.
| TL;DR: On 12 hard coding tasks, Opus alone solved 81% at $1.20/task. Opus + DeepSeek V4.1 Flash solved 67% at $1.29. I tried 7 team designs (41 setups, ~1,200 runs). None beat Opus alone, and most cost more. Why I did this. Opus is great but expensive. I was looking for one of two things: a team that beats a single model without a big jump in price, or a setup that's much cheaper than Opus and almost as good. So I had Opus plan and review while cheaper models wrote the code. The tasks. I used to build small benchmarks from my own agent sessions, but those tasks turned out too easy. This time I picked 12 hard tasks that look like my daily work, from TB4, DeepSWE, SWE-rebench, SWE-bench Pro V2 and Longtail-SWE. Hidden tests decide pass or fail, and every setup ran each task 3 times. I also went through the runs by hand: fixed broken tests or prompts in 9 tasks, and gave zero to "solutions" that just downloaded the real fix from upstream. The chart shows the main setups. The best Opus-led team got 69%. Single cheaper models beat every team: GPT-6.1 Terra got 78% and Sol 75%, both at $0.17/task. A few things stood out: Cheap models did get better with a reviewer. Having Sol review their code added 8–11 points for about $0.07/task. But it only worked one way: Sol made Opus's solutions better, while Opus reviewing Sol made things worse. It's only 12 tasks, so small differences are noise, and shorter tasks may look different. Next, I'm testing a 1-hour cache for Opus. Full table, every setup with scores, confidence intervals, cost and time per task. https://ibragim.dev/leaderboard-v2/ If you know a setup that should beat Opus alone, tell me how you'd build it. I'll run it and add it to the table. submitted by /u/Fabulous_Pollution10 [link] [comments] |
#Claude Opus 5.5#DeepSeek V4.1 Flash#GPT-6.1 Terra#GPT-6.1 Sol#TB4#DeepSWE#SWE-rebench#SWE-bench Pro V2#Longtail-SWE