We changed requirements in a coding benchmark and found models talking themselves out of following them.

Sometimes the agent understood the new requirement perfectly. It described what needed to change, considered an implementation, and then decided the hidden tests probably wanted something else. In one run, a model implemented the change correctly, asked another model for the benchmark’s original answer, and removed its own correct code after receiving it.

This is an awkward habit for a coding assistant to have. It is also a habit that a familiar benchmark can reward. If the model has guessed what the tests expect, it gets the same passing result as an agent that worked carefully from the requirements.

We built MaxxBench to examine how much coding performance depends on those expectations. It changes the instructions or starting code of 89 DeepSWE tasks, giving models familiar engineering work with a reason to revise their usual approach.

Methodology: Small mutations...

Once a benchmark becomes popular, it becomes part of the development process. Its tasks can enter training data. Its scores influence checkpoint selection. Developers adjust prompts and agents against its failures. There are plenty of opportunities to get better at that particular exam. We call gains tied to its particular tasks and conventions benchmaxxing.