TestPrism evaluates agent-written tests against many implementations
A new arXiv preprint introduces TestPrism, a benchmark of 300 test tasks drawn from 17 sources plus 3000 candidate implementations split evenly between valid and invalid solutions, designed to judge tests generated by LLM coding agents. Its main metric, Joint Success Function, requires a generated test to fail on the initial program state, accept every valid candidate and reject every invalid one; across fourteen baseline coding-agent configurations it reaches only 28.00%, versus 59.67% for single-reference success. The authors also propose TestHelix, which combines heterogeneous synthesis of test–repair pairs with peer cross validation and recursive self improvement (RSI), and report gains of 8.67 to 9.00 percentage points over native harness comparators across two models.
It suggests scoring agent-written tests against one reference solution overstates quality, and offers a stricter multi-implementation benchmark.
Computer Science > Software Engineering
arXiv:2610.12289 (cs)
[Submitted on 8 Oct 2026]
Title:TestPrism: Rethinking Test Evaluation Beyond a Single Reference
Authors:Han Li, Lingxiang Hu, Jiacheng Huang, Ziqian Jiang, Jingkai Luo, Wei Gao, Yunfan Tan, Zun Wang, Jiaheng Liu
View a PDF of the paper titled TestPrism: Rethinking Test Evaluation Beyond a Single Reference, by Han Li and 8 other authors