Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

VeriCUA verifies computer-use agents by gathering evidence

AI summary

Google and the University of Hong Kong introduced VeriCUA, a validation process that surveys the environment, defines acceptance criteria, gathers GUI and code-based evidence, and applies deterministic evidence rules. The paper reports leading results across five benchmarks, including 90.5% accuracy on AndroidWorld and a 5.4% false-positive rate on ScienceBoard. It also reports rejecting roughly 80% of agents’ false completion claims.

In-depth insight

VeriCUA offers a concrete approach to reducing false-positive task completions in agent evaluation, reinforcement-learning rewards, and training-data curation, although the reported evaluations use only Gemini-family backbones and the code was not released.

💡 一句话总结:Google 和港大的 VeriCUA 把「判定计算机操作智能体是否真完成任务」从一次模型调用打分,升级成一套会主动取证的验证流程——先勘察环境再定验收标准,GUI 和代码两条通道找证据,按证据可信度规则仲裁。五个基准全部领先,假阳性率最多砍到基线的四分之一。做 agent 评测、RL 奖励、数据筛选的人都该看看。

先问一个可能戳到你的问题:你训 agent 的时候,有没有怀疑过训练集里那些「成功」轨迹?

比如你让智能体「把这段笔记保存到备忘录」,它操作了一通,界面上明明显示着笔记内容,于是它报告:任务完成。轨迹看着没毛病,验证器打了个勾,这条样本就进了你的 RL 奖励池或者 SFT 数据集。

#Google#University of Hong Kong#VeriCUA#Gemini#AndroidWorld#ScienceBoard#OSWorld#OSReward#OS-Themis

Original · 9 min readRead the original
Group chat

No comments yet. Start the conversation.