Parsimony Benchmark Measures AI Coding Complexity
Reddit user lasushin released Parsimony, an open-source benchmark that counts syntax units in AI-generated patches and compares them with human fixes for the same issues. Its initial analysis reports a median of about 1.9 times the human-fix code on SWE-bench Verified and about one times it on DeepSWE, while harness comparisons using the same two open-weight models ranged from 0.80 times for Codex to 0.99 times for mini-SWE-agent and OpenClaw.
Parsimony adds a code-complexity measure to benchmarks that primarily track issue resolution, token use, or cost, making excess code and potential overengineering directly comparable with human patches.
| Existing benchmarks measure the solve rate and token output/cost, but don't punish excess code or overengineering, which increases technical debt in the long term. Parsimony counts syntax units (statements, calls, names) in the patches other benchmarks publish. It doesn't count lines, comments, tests. Each model or harness is compared with the human fix for the same issues it solved. - Smarter models surprisingly write slightly more code. - Models: the median model adds 1.9× the human fix on SWE-bench Verified (older models), but about 1× on DeepSWE. - Harnesses: with the same two open-weight models, the harness alone moves it from 0.80× (Codex) to 0.99× (mini-SWE-agent, OpenClaw). The harness data comes from Joel Niklaus's Harness Leaderboard. https://parsimony.lasu.dev https://github.com/La5u/Parsimony Looking for: public patches for newer models, any ideas or contributors for self run models. * some newer models are self run in codex and claude code, shown as faded (unverified) in DeepSWE submitted by /u/lasushin [link] [comments] |
#Parsimony#SWE-bench Verified#DeepSWE#Codex#mini-SWE-agent#OpenClaw