Opus主导编码团队未改善性价比
AI 总结
Reddit作者/u/Fabulous_Pollution10在12项高难度编码任务中报告,Claude Opus 5.5单独使用可解决81%的任务,每题成本1.20美元;搭配DeepSeek V4.1 Flash的通过率为67%,每题成本1.29美元。作者测试7种团队设计、41种配置和约1,200次运行,结果均未超过Opus单独使用,且多数成本更高。
深度见解
实验显示,让昂贵模型负责规划和审查、再由低成本模型编写代码,未必能改善AI编码的成本效益;但任务样本较小,微小性能差异仍可能受到噪声影响。
| TL;DR: On 12 hard coding tasks, Opus alone solved 81% at $1.20/task. Opus + DeepSeek V4.1 Flash solved 67% at $1.29. I tried 7 team designs (41 setups, ~1,200 runs). None beat Opus alone, and most cost more. Why I did this. Opus is great but expensive. I was looking for one of two things: a team that beats a single model without a big jump in price, or a setup that's much cheaper than Opus and almost as good. So I had Opus plan and review while cheaper models wrote the code. The tasks. I used to build small benchmarks from my own agent sessions, but those tasks turned out too easy. This time I picked 12 hard tasks that look like my daily work, from TB4, DeepSWE, SWE-rebench, SWE-bench Pro V2 and Longtail-SWE. Hidden tests decide pass or fail, and every setup ran each task 3 times. I also went through the runs by hand: fixed broken tests or prompts in 9 tasks, and gave zero to "solutions" that just downloaded the real fix from upstream. The chart shows the main setups. The best Opus-led team got 69%. Single cheaper models beat every team: GPT-6.1 Terra got 78% and Sol 75%, both at $0.17/task. A few things stood out: Cheap models did get better with a reviewer. Having Sol review their code added 8–11 points for about $0.07/task. But it only worked one way: Sol made Opus's solutions better, while Opus reviewing Sol made things worse. It's only 12 tasks, so small differences are noise, and shorter tasks may look different. Next, I'm testing a 1-hour cache for Opus. Full table, every setup with scores, confidence intervals, cost and time per task. https://ibragim.dev/leaderboard-v2/ If you know a setup that should beat Opus alone, tell me how you'd build it. I'll run it and add it to the table. submitted by /u/Fabulous_Pollution10 [link] [comments] |
#Claude Opus 5.5#DeepSeek V4.1 Flash#GPT-6.1 Terra#GPT-6.1 Sol#TB4#DeepSWE#SWE-rebench#SWE-bench Pro V2#Longtail-SWE