VeriCUA verifies computer-use agents by gathering evidence
Google and the University of Hong Kong introduced VeriCUA, a validation process that surveys the environment, defines acceptance criteria, gathers GUI and code-based evidence, and applies deterministic evidence rules. The paper reports leading results across five benchmarks, including 90.5% accuracy on AndroidWorld and a 5.4% false-positive rate on ScienceBoard. It also reports rejecting roughly 80% of agents’ false completion claims.
VeriCUA offers a concrete approach to reducing false-positive task completions in agent evaluation, reinforcement-learning rewards, and training-data curation, although the reported evaluations use only Gemini-family backbones and the code was not released.
💡 一句话总结:Google 和港大的 VeriCUA 把「判定计算机操作智能体是否真完成任务」从一次模型调用打分,升级成一套会主动取证的验证流程——先勘察环境再定验收标准,GUI 和代码两条通道找证据,按证据可信度规则仲裁。五个基准全部领先,假阳性率最多砍到基线的四分之一。做 agent 评测、RL 奖励、数据筛选的人都该看看。
先问一个可能戳到你的问题:你训 agent 的时候,有没有怀疑过训练集里那些「成功」轨迹?
比如你让智能体「把这段笔记保存到备忘录」,它操作了一通,界面上明明显示着笔记内容,于是它报告:任务完成。轨迹看着没毛病,验证器打了个勾,这条样本就进了你的 RL 奖励池或者 SFT 数据集。
#Google#University of Hong Kong#VeriCUA#Gemini#AndroidWorld#ScienceBoard#OSWorld#OSReward#OS-Themis