创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

Parsimony衡量AI代码复杂度

AI 总结

Reddit用户lasushin发布开源基准Parsimony,通过统计AI补丁中的语法单元,并将其与解决相同问题的人工修复进行比较。其初步分析显示,模型在SWE-bench Verified上的代码量中位数为人工修复的约1.9倍,在DeepSWE上约为1倍;使用相同两个开放权重模型的执行框架对比则从Codex的0.80倍到mini-SWE-agent和OpenClaw的0.99倍。

深度见解

Parsimony为主要关注解题率、Token用量或成本的基准补充了代码复杂度指标,使冗余代码和潜在过度工程能够与人工补丁直接比较。

Existing benchmarks measure the solve rate and token output/cost, but don't punish excess code or overengineering, which increases technical debt in the long term. Parsimony counts syntax units (statements, calls, names) in the patches other benchmarks publish. It doesn't count lines, comments, tests. Each model or harness is compared with the human fix for the same issues it solved. - Smarter models surprisingly write slightly more code. - Models: the median model adds 1.9× the human fix on SWE-bench Verified (older models), but about 1× on DeepSWE. - Harnesses: with the same two open-weight models, the harness alone moves it from 0.80× (Codex) to 0.99× (mini-SWE-agent, OpenClaw). The harness data comes from Joel Niklaus's Harness Leaderboard. https://parsimony.lasu.dev https://github.com/La5u/Parsimony Looking for: public patches for newer models, any ideas or contributors for self run models. * some newer models are self run in codex and claude code, shown as faded (unverified) in DeepSWE submitted by /u/lasushin [link] [comments]

#Parsimony#SWE-bench Verified#DeepSWE#Codex#mini-SWE-agent#OpenClaw

原文约 1 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。