SWE-Pair launches benchmark for fixing two bugs at once
SWE-Pair benchmark authors say the benchmark contains 161 tasks drawn from 69 Python projects, with each task requiring a coding agent to fix two real bugs simultaneously. In their reported results, DeepSeek V4.1 Flash at medium effort solved 90% of tasks at about six cents per solved task, while GPT-6 Luna at maximum effort solved 78% at about 23 cents. DeepSeek at maximum effort scored 88% and cost 1.7 times more than its medium-effort setup. These figures come from a single social-media report by the authors and were not independently confirmed in the supplied evidence.
The benchmark provides a cost-and-performance comparison for coding agents, while its two-bug format tests whether agents can handle two real fixes in one task.
| We built a benchmark of 161 tasks from 69 Python projects where the agent has to fix two real bugs at once. The cheapest setup came out on top. DeepSeek V4.1 Flash at medium effort solved 90% of tasks, at about 6 cents per solved task. GPT-6 Luna at max effort solved 78%, at about 23 cents. More reasoning didn't help either. DeepSeek at max effort scored 88%, slightly below medium, and cost 1.7x more. https://preview.redd.it/8ksnyprzst0i1.png?width=1200&format=png&auto=webp&s=e05f88446be40b85ceddf9bf2e9796e57efc9314 Results and every agent run: https://labs.evaligo.com/swe-pair Tasks: https://huggingface.co/datasets/evaligo/swe-pair To run your own agent on it, it's one Harbor command: harbor run -d evaligo/swe-pair -a <agent> -m <provider/model> submitted by /u/heyitsdannyle [link] [comments] |