Opus 5.5 test challenges SlopCodeBench's code-messiness penalty
A Reddit author ran Opus 5.5 across all 196 SlopCodeBench checkpoints, finding that erosion and verbosity increased in 22 of 35 problems. Messiness did not predict whether the next checkpoint was solved, while about 0.3% of previously passing tests regressed and messier repositories did not regress more often; codebase size, rather than messiness, was associated with higher costs, with the next checkpoint costing about 19% more each time the codebase doubled. The author also reported that cleaner code on the leaderboard had essentially no correlation with solving more problems, while Opus 5.5 passed 84% to 92% of tests for each new specification.
The test provides a substantive counterpoint to SlopCodeBench's warning that increasingly messy agent code directly undermines subsequent work, but its conclusions are limited to one model, small programs and a single observational run.
A few months ago, I ran across this paper proposing a benchmark called SlopCodeBench. The idea is that you give an agent a problem split into a bunch of checkpoints, and at each one it has to add a new feature on top of its own old code without breaking anything. Along the way the benchmark measures how "sloppy" the code gets, using two metrics: structural erosion (how much complexity piles up in a few giant branchy functions) and verbosity (how much of the code is redundant).
The paper found that agent code gets messier the longer the agent works on it, and that agents get worse at keeping everything passing as they go. The obvious takeaway is that the agent is digging its own grave. Messy code is hard to extend, so it fails.
I wasn't convinced that held for agents, so I tested it. I ran Opus 5.5 through all 196 checkpoints ($266 in API costs, ouch) and checked whether messier code predicted worse results.
Short version:
- Its code did get messier, just like the paper says. Erosion and verbosity both went up in 22 of 35 problems.