Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Opus-led coding teams did not improve cost-performance

AI summary

In a Reddit experiment by author /u/Fabulous_Pollution10, Claude Opus 5.5 alone solved 81% of 12 hard coding tasks at $1.20 per task, while pairing it with DeepSeek V4.1 Flash produced 67% at $1.29. Across seven team designs, 41 setups and roughly 1,200 runs, none beat Opus alone and most cost more; the author attributes some failures to weak workers, handoff errors and prompt-cache rebuilding.

In-depth insight

The reported results suggest that simply assigning planning and review to an expensive model while delegating implementation to cheaper models may not improve AI coding economics, although the small task sample limits confidence in small performance differences.

TL;DR: On 12 hard coding tasks, Opus alone solved 81% at $1.20/task. Opus + DeepSeek V4.1 Flash solved 67% at $1.29. I tried 7 team designs (41 setups, ~1,200 runs). None beat Opus alone, and most cost more. Why I did this. Opus is great but expensive. I was looking for one of two things: a team that beats a single model without a big jump in price, or a setup that's much cheaper than Opus and almost as good. So I had Opus plan and review while cheaper models wrote the code. The tasks. I used to build small benchmarks from my own agent sessions, but those tasks turned out too easy. This time I picked 12 hard tasks that look like my daily work, from TB4, DeepSWE, SWE-rebench, SWE-bench Pro V2 and Longtail-SWE. Hidden tests decide pass or fail, and every setup ran each task 3 times. I also went through the runs by hand: fixed broken tests or prompts in 9 tasks, and gave zero to "solutions" that just downloaded the real fix from upstream. The chart shows the main setups. The best Opus-led team got 69%. Single cheaper models beat every team: GPT-6.1 Terra got 78% and Sol 75%, both at $0.17/task. A few things stood out: Cheap models did get better with a reviewer. Having Sol review their code added 8–11 points for about $0.07/task. But it only worked one way: Sol made Opus's solutions better, while Opus reviewing Sol made things worse. It's only 12 tasks, so small differences are noise, and shorter tasks may look different. Next, I'm testing a 1-hour cache for Opus. Full table, every setup with scores, confidence intervals, cost and time per task. https://ibragim.dev/leaderboard-v2/ If you know a setup that should beat Opus alone, tell me how you'd build it. I'll run it and add it to the table. submitted by /u/Fabulous_Pollution10 [link] [comments]

#Claude Opus 5.5#DeepSeek V4.1 Flash#GPT-6.1 Terra#GPT-6.1 Sol#TB4#DeepSWE#SWE-rebench#SWE-bench Pro V2#Longtail-SWE

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.