DeepSeek v4.1 flash tops 1% on SWE-sweep bug-fixing benchmark
AI summary
The SWE-sweep evaluators reported that DeepSeek v4.1 flash (max) was the first open-weight model they evaluated to exceed 1% on the benchmark. The test gives models a potentially large repository containing hundreds of bugs, without identifying the defects, and evaluates them in a shared harness to isolate model capability. The evaluation code was released for others to run.
In-depth insight
DeepSeek v4.1 flash (max) crossing 1% provides a current open-weight reference point on a demanding repository-level bug-fixing task, while the shared harness and published evaluation code make the result more open to verification.
| Hi, I've just updated the SWE-sweep leaderboard with more open-weight models! DeepSeek v4.1 flash (max) is the first open-weight model we evaluated that gets >1%. This benchmark is extremely challenging: Models get a snapshot of a (potentially very large) repo containing hundreds of bugs. They don't receive any hint about what these bugs are and need to fix them all. This makes for a very reasoning-heavy, long-horizon task. To isolate just the model capabilities we also evaluate all models in the same harness (but we did do some comparisons with Claude Code and Codex harnesses and neither seemed to help significantly). However, it is possibly to scale things up somewhat with multiagents. You can run the eval yourself (code on github) The best proprietary model at the moment is Gemini 4 Hydrogen with 13.5% (according to google's model card), followed by GPT Astra 6 with 9.5% (evaluated by us). So open-weight models still have a long way to go. Still extending the open-source leaderboard. What other models should we add? submitted by /u/klieret [link] [comments] |
#DeepSeek v4.1 flash (max)#SWE-sweep#Claude Code#Codex#Gemini 4 Hydrogen#GPT Astra 6