PaperBenchX: best AI agent fully reproduces only 14% of papers
UniPat AI's PaperBenchX benchmark gives 10 frontier agent configurations a published paper and asks them to re-run the key experiments in real scientific software, across 93 tasks covering 12 research directions. According to the team's published results, 70 of the 93 tasks were not fully reproduced by any configuration, and the top performer, GPT-6 Astra, fully cleared only 14% (13 tasks), while Chinese models clustered around 7% — roughly half that. Scoring separates modeling, execution and scientific verification across 3,168 scorable criteria, with a 4–24 hour per-task budget and outputs wiped and re-run in an isolated environment before grading.
Reproducibility is the groundwork for letting AI agents run research, and this benchmark turns "can AI reliably reproduce existing work" into something that can be checked item by item.