Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Parsimony Benchmark Measures AI Coding Complexity

AI summary

Reddit user lasushin released Parsimony, an open-source benchmark that counts syntax units in AI-generated patches and compares them with human fixes for the same issues. Its initial analysis reports a median of about 1.9 times the human-fix code on SWE-bench Verified and about one times it on DeepSWE, while harness comparisons using the same two open-weight models ranged from 0.80 times for Codex to 0.99 times for mini-SWE-agent and OpenClaw.

In-depth insight

Parsimony adds a code-complexity measure to benchmarks that primarily track issue resolution, token use, or cost, making excess code and potential overengineering directly comparable with human patches.

Existing benchmarks measure the solve rate and token output/cost, but don't punish excess code or overengineering, which increases technical debt in the long term. Parsimony counts syntax units (statements, calls, names) in the patches other benchmarks publish. It doesn't count lines, comments, tests. Each model or harness is compared with the human fix for the same issues it solved. - Smarter models surprisingly write slightly more code. - Models: the median model adds 1.9× the human fix on SWE-bench Verified (older models), but about 1× on DeepSWE. - Harnesses: with the same two open-weight models, the harness alone moves it from 0.80× (Codex) to 0.99× (mini-SWE-agent, OpenClaw). The harness data comes from Joel Niklaus's Harness Leaderboard. https://parsimony.lasu.dev https://github.com/La5u/Parsimony Looking for: public patches for newer models, any ideas or contributors for self run models. * some newer models are self run in codex and claude code, shown as faded (unverified) in DeepSWE submitted by /u/lasushin [link] [comments]

#Parsimony#SWE-bench Verified#DeepSWE#Codex#mini-SWE-agent#OpenClaw

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.