TestPrism
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 2
- TestPrism Exposes Coding Agent Test Reliability Gaps
南京大学刘佳恒团队提出测试生成基准 TestPrism,包含来自17个来源的300个测试任务和3000个候选实现。其核心指标 Joint Success Function(JSF)要求测试在原始代码上失败、接受所有正确实现并拒绝所有错误实现;报道显示,14种 Coding Agent 配置中最佳 JSF 仅28.00%。研究还分析了286条失败轨迹,并提出通过异构合成、同伴交叉验证和递归自我改进提升结果的 TestHelix 框架。
36氪 人工智能 · 🔥 0 - TestPrism evaluates agent-written tests against many implementations
A new arXiv preprint introduces TestPrism, a benchmark of 300 test tasks drawn from 17 sources plus 3000 candidate implementations split evenly between valid and invalid solutions, designed to judge tests generated by LLM coding agents. Its main metric, Joint Success Function, requires a generated test to fail on the initial program state, accept every valid candidate and reject every invalid one; across fourteen baseline coding-agent configurations it reaches only 28.00%, versus 59.67% for single-reference success. The authors also propose TestHelix, which combines heterogeneous synthesis of test–repair pairs with peer cross validation and recursive self improvement (RSI), and report gains of 8.67 to 9.00 percentage points over native harness comparators across two models.
Hugging Face · Papers · 🔥 0
Experience and discussion from the community
Share my TestPrism experienceAsk about TestPrism
Nobody has shared their experience with TestPrism yet.