DeepSeek v4.1 flash突破SWE-sweep 1%
AI 总结
SWE-sweep评测方称，DeepSeek v4.1 flash（max）是其测试的首个得分超过1%的开放权重模型。该基准让模型处理可能很大的代码仓库，其中包含数百个缺陷，但不提示具体缺陷位置；所有模型均在同一测试框架下运行，以尽量隔离模型能力差异。评测代码已公开，供其他人运行。
深度见解
DeepSeek v4.1 flash（max）突破1%，为高难度仓库级批量修bug任务提供了开放权重模型参考；统一测试框架和公开评测代码也有助于复核结果。
| Hi, I've just updated the SWE-sweep leaderboard with more open-weight models! DeepSeek v4.1 flash (max) is the first open-weight model we evaluated that gets >1%. This benchmark is extremely challenging: Models get a snapshot of a (potentially very large) repo containing hundreds of bugs. They don't receive any hint about what these bugs are and need to fix them all. This makes for a very reasoning-heavy, long-horizon task. To isolate just the model capabilities we also evaluate all models in the same harness (but we did do some comparisons with Claude Code and Codex harnesses and neither seemed to help significantly). However, it is possibly to scale things up somewhat with multiagents. You can run the eval yourself (code on github) The best proprietary model at the moment is Gemini 4 Hydrogen with 13.5% (according to google's model card), followed by GPT Astra 6 with 9.5% (evaluated by us). So open-weight models still have a long way to go. Still extending the open-source leaderboard. What other models should we add? submitted by /u/klieret [link] [comments] |
#DeepSeek v4.1 flash (max)#SWE-sweep#Claude Code#Codex#Gemini 4 Hydrogen#GPT Astra 6