Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Stanford study finds Terminal-Bench 2.0 defects alter agent scores

AI summary

A Stanford study summarized by 掘金 found that 28 of 89 tasks in Terminal-Bench 2.0 needed fixes, including 14 newly identified defects, and the repairs shipped in Terminal-Bench 2.1. Re-evaluating 19 agents produced aggregate accuracy more than 10 percentage points higher after the fixes, with some agents moving three ranking positions. The study also found that timeouts accounted for more than half of failed trials under the default budget, while extending the budget nearly erased a 20-point gap between harnesses.

In-depth insight

The findings show that benchmark version, harness and time budget can materially affect AI agent comparisons, even when the broad performance ranking remains stable.

TL;DR:斯坦福 Ré 组的最新工作把「基准测试会像软件一样腐烂」变成了一个可测量、可维护的工程问题——他们提出基准维护债务概念,用「智能体审计 + 人工把关」的持续验证流程给 Terminal-Bench 2.0 做体检,结果约 30% 的任务带缺陷;修复版 2.1 让 19 个智能体的整体准确率上涨超 10 个百分点,个别模型榜位移动 3 名。更扎心的是:一半以上的失败试验其实是超时,把时间预算拉长后,20 个百分点的 harness 差距几乎消失。

先问一个可能戳到你的问题:你上个月盯着的那个 Agent 榜单,今天还准吗?

做 agent 的朋友对这套流程太熟了:新模型发布 → 官网贴出 SWE-Bench / Terminal-Bench 分数 → 你默默把自己的 agent 和它对齐 → 榜单成为 roadmap。但很少有人想过:那个基准本身,可能已经悄悄烂掉了。发布时千锤百炼的任务,几个月后可能因为依赖的包升级了、API 改版了、网页下线了而跑不起来;更隐蔽的是,模型变聪明之后会找到基准设计者当年没想到的合法解法——然后被一个过时的验证器判负。

#Terminal-Bench 2.0#Terminal-Bench 2.1#Stanford University

Original · 8 min readRead the original
Group chat

No comments yet. Start the conversation.