Learnings from Auditing Public Evaluations

Evaluations are the foundation of LLM training, allowing us to put concrete numbers on the capabilities we care about. Public evaluations allow us to build a shared understanding of these capabilities and how different models perform on them, thus providing important insights. They also allow everyone in the community to scrutinize these benchmarks and propose changes.

While running popular evaluations reported in model cards, we ran into several issues that affect how results should be interpreted. After applying fixes, some of the benchmarks can be considered saturated. Below, we showcase some of them and discuss the design patterns and reward hacks we observed while running frontier models. We also open-source our fixes for the community to make the evaluation landscape more robust.

AutomationBench

AutomationBench is Zapier’s benchmark for agents completing business workflows that span multiple applications through APIs. Agents have to find the right endpoints themselves and follow business rules.