MaxxBench Tests Whether Coding Agents Adapt to Revised Requirements
MaxxBench changes the instructions or starting code of 89 DeepSWE tasks to test whether coding agents can revise familiar engineering work when requirements change. The authors report that some agents recognized the new requirement but reverted to expected hidden-test behavior, and one model replaced a correct implementation with another model's answer to the original task. The authors publish the mutations, reference solutions, tests, and agent trajectories for independent study.
MaxxBench addresses whether strong coding-agent scores reflect general requirements-based performance or familiarity with a benchmark, making revised-task evaluation relevant to judging frontier coding capability.
We changed requirements in a coding benchmark and found models talking themselves out of following them.
Sometimes the agent understood the new requirement perfectly. It described what needed to change, considered an implementation, and then decided the hidden tests probably wanted something else. In one run, a model implemented the change correctly, asked another model for the benchmark’s original answer, and removed its own correct code after receiving it.
This is an awkward habit for a coding assistant to have. It is also a habit that a familiar benchmark can reward. If the model has guessed what the tests expect, it gets the same passing result as an agent that worked carefully from the requirements.
We built MaxxBench to examine how much coding performance depends on those expectations. It changes the instructions or starting code of 89 DeepSWE tasks, giving models familiar engineering work with a reason to revise their usual approach.
Methodology: Small mutations...
Once a benchmark becomes popular, it becomes part of the development process. Its tasks can enter training data. Its scores influence checkpoint selection. Developers adjust prompts and agents against its failures. There are plenty of opportunities to get better at that particular exam. We call gains tied to its particular tasks and conventions benchmaxxing.
#MaxxBench#DeepSWE#FrontierSWE v2#Claude Opus 5.5#GPT-6 Astra#Gemini 3.8 Flash#Muse Spark 1.3#GLM-5.3