Parsimony衡量AI代码复杂度
AI 总结
Reddit用户lasushin发布开源基准Parsimony,通过统计AI补丁中的语法单元,并将其与解决相同问题的人工修复进行比较。其初步分析显示,模型在SWE-bench Verified上的代码量中位数为人工修复的约1.9倍,在DeepSWE上约为1倍;使用相同两个开放权重模型的执行框架对比则从Codex的0.80倍到mini-SWE-agent和OpenClaw的0.99倍。
深度见解
Parsimony为主要关注解题率、Token用量或成本的基准补充了代码复杂度指标,使冗余代码和潜在过度工程能够与人工补丁直接比较。
| Existing benchmarks measure the solve rate and token output/cost, but don't punish excess code or overengineering, which increases technical debt in the long term. Parsimony counts syntax units (statements, calls, names) in the patches other benchmarks publish. It doesn't count lines, comments, tests. Each model or harness is compared with the human fix for the same issues it solved. - Smarter models surprisingly write slightly more code. - Models: the median model adds 1.9× the human fix on SWE-bench Verified (older models), but about 1× on DeepSWE. - Harnesses: with the same two open-weight models, the harness alone moves it from 0.80× (Codex) to 0.99× (mini-SWE-agent, OpenClaw). The harness data comes from Joel Niklaus's Harness Leaderboard. https://parsimony.lasu.dev https://github.com/La5u/Parsimony Looking for: public patches for newer models, any ideas or contributors for self run models. * some newer models are self run in codex and claude code, shown as faded (unverified) in DeepSWE submitted by /u/lasushin [link] [comments] |
#Parsimony#SWE-bench Verified#DeepSWE#Codex#mini-SWE-agent#OpenClaw