A6000 Agent Evaluation’s Green Result May Be Simulated
A Juejin AI analysis found that a historical A6000 Agent evaluation script labeled a run as “3 passed in 0.04s” after finding “Bearer” in source text, without starting a test process. The author reproduced a local counterexample in which adding only a comment changed the synthetic verdict from failed to green; this was not a model benchmark, pytest run, or replay of historical sessions. The report says tool calls and patches could still have occurred, but the green flag alone does not establish that the target program passed.
The reproduced counterexample shows that the cited success signal can change without target behavior or tests changing, making the provenance of Agent evaluation results essential before they enter task-success statistics.
Agent 说“3 passed”就算修好了?先检查是谁判的通过
一份 A6000 评测文档把工具循环描述为“真实 debug”,并展示了 all_green=True。但对应脚本里的 run_tests 没有启动测试进程:它检查一段源码文本中是否出现 Bearer,满足条件就返回固定的 3 passed in 0.04s。
这个差别会改变整条结论。 工具调用 可能真实发生,模型也可能认真生成了补丁,但“工具返回了通过”不等于“被测程序通过了测试”。要把这样的记录纳入 Agent 任务成功率,首先需要确认:最终判定观察的是目标程序的行为,还是模拟器里预设的字符串?
本文回查既有素材,并实际运行一个隔离反例。没有重新连接 A6000、调用模型或执行历史会话;文中的新结果只属于本地判定函数复算,不是模型跑分。