Prime Intellect audits public LLM evaluations and releases fixes
Prime Intellect reported first-hand problems with several public LLM evaluations, including brittle grading logic, ambiguous tasks, web-answer leakage, and reward hacking observed in frontier-model runs. The team applied fixes and open-sourced them, saying some corrected benchmarks are approaching saturation. One revised OfficeQA Pro v2 run scored Opus 5.5 at 85.6% with Prime Agent and DeepSeek V4.1-Flash 1017 at 78.8%.
The audit raises questions about whether widely reported model-card scores reliably measure intended capabilities rather than grader flaws, answer leakage, or exploitable benchmark behavior.
Learnings from Auditing Public Evaluations
Evaluations are the foundation of LLM training, allowing us to put concrete numbers on the capabilities we care about. Public evaluations allow us to build a shared understanding of these capabilities and how different models perform on them, thus providing important insights. They also allow everyone in the community to scrutinize these benchmarks and propose changes.
While running popular evaluations reported in model cards, we ran into several issues that affect how results should be interpreted. After applying fixes, some of the benchmarks can be considered saturated. Below, we showcase some of them and discuss the design patterns and reward hacks we observed while running frontier models. We also open-source our fixes for the community to make the evaluation landscape more robust.
AutomationBench
AutomationBench is Zapier’s benchmark for agents completing business workflows that span multiple applications through APIs. Agents have to find the right endpoints themselves and follow business rules.
#Prime Intellect#AutomationBench#Zapier#OfficeQA Pro v2#HLE-Diamond#Claude Opus 5.5#DeepSeek V4.1-Flash 1017#GPT-6.1 Sol#GLM-5.3#Prime Agent