Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

SWE-Pair launches benchmark for fixing two bugs at once

AI summary

SWE-Pair benchmark authors say the benchmark contains 161 tasks drawn from 69 Python projects, with each task requiring a coding agent to fix two real bugs simultaneously. In their reported results, DeepSeek V4.1 Flash at medium effort solved 90% of tasks at about six cents per solved task, while GPT-6 Luna at maximum effort solved 78% at about 23 cents. DeepSeek at maximum effort scored 88% and cost 1.7 times more than its medium-effort setup. These figures come from a single social-media report by the authors and were not independently confirmed in the supplied evidence.

In-depth insight

The benchmark provides a cost-and-performance comparison for coding agents, while its two-bug format tests whether agents can handle two real fixes in one task.

We built a benchmark of 161 tasks from 69 Python projects where the agent has to fix two real bugs at once. The cheapest setup came out on top. DeepSeek V4.1 Flash at medium effort solved 90% of tasks, at about 6 cents per solved task. GPT-6 Luna at max effort solved 78%, at about 23 cents. More reasoning didn't help either. DeepSeek at max effort scored 88%, slightly below medium, and cost 1.7x more. https://preview.redd.it/8ksnyprzst0i1.png?width=1200&format=png&auto=webp&s=e05f88446be40b85ceddf9bf2e9796e57efc9314 Results and every agent run: https://labs.evaligo.com/swe-pair Tasks: https://huggingface.co/datasets/evaligo/swe-pair To run your own agent on it, it's one Harbor command: harbor run -d evaligo/swe-pair -a <agent> -m <provider/model> submitted by /u/heyitsdannyle [link] [comments]

#SWE-Pair#DeepSeek V4.1 Flash#GPT-6 Luna#Harbor#Evaligo

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.