Audit Agent Leaderboards Before Trusting the Scores
A 200-line Python script uses Wilson score intervals, two-proportion z-tests and minimum detectable effect calculations to audit AI agent leaderboard results. Under the article's assumed sample size of 500 tasks, Claude Opus 5.5's 90.4% lead over GPT-6 Astra is not statistically significant, while the article also reviews reported benchmark defects and cheating cases.
The article provides a practical method for checking whether small AI agent score gaps are meaningful and emphasizes that benchmark quality, task count, scaffold and test version must be reported before rankings can support comparisons.
一个 200 行的脚本,教你在信任何 Agent 榜单之前先算三件事:区间多宽、差显不显著、要多少题才够。
背景:这周榜单圈连着塌了三次
先把事实摆齐,全部可查:
| 时间 | 事件 |
|---|---|
| 11-02 | Agent Index v1.2 预览:七项全有才计分,86 个模型仅 26 个合格,Opus 5.5 90.4 居首 |
| 11-03 | SemiAnalysis GDPx 访谈:Terminal-Bench 4 里约六成任务是坏的 |
| 11-05 | 有人指出 Terminal-Bench 4.0.0 早在 9-04 就被 Epoch 判 Flawed(门槛:缺陷影响 ≥20% 总分) |
| 11-09 | Terminal-Bench 2.0 前三名全部因作弊被移除 |
Epoch 复审 21 个基准的结果是 10 Flawed / 5 Verified / 6 Not Enough Info。作弊细节也很具体:ForgeCode 靠 AGENTS.md 塞答案拿 81.8%,去掉后 71.7%、掉到第 14;Pilot 82.9% 的成绩里 429 条轨迹有 415 条第一步是 cat /tests/test_outputs.py。
这些是"评测设计"层面的问题。但还有一层更基础、更容易被工程师忽略的问题:就算榜单完全干净,你看到的那个分数,本身也带着采样误差。
#score_audit.py#Agent Index v1.2#Terminal-Bench 4.0.0#Claude Opus 5.5#GPT-6 Astra#Codex CLI#Terminus-2