GUI Agents Face Reliability Limits in the Accessibility Layer
A report argues that screenshot-free GUI agents depend on stable, uniquely named nodes in the operating system’s accessibility tree. Its audit of two interface forms found unnamed-node rates of 0% and 50%. In a simulated tree change, 70% of reference positions pointed to different controls. The report also notes that agent-desktop’s reported 55 consecutive successful actions correspond to a 6.5% upper bound on the failure rate at 95% confidence under the stated statistical assumptions, rather than establishing a failure rate below 1%.
The report provides engineering guidance for GUI-agent developers: audit unnamed and duplicate controls, refresh accessibility-tree snapshots after each action, and validate outcomes through interface-state assertions rather than trusting the agent’s completion claims.
今天热榜上一天冒出四条 GUI Agent:agent-desktop 连点 55 次 0 失误、Mika 在安卓后台虚拟屏上替你操作 App、WorkBuddy 接管浏览器,还有一篇讲 App 未来会被 Agent 以三种方式调用。
全在回答同一个问题:这件事能不能做。
第二个问题没人问:在什么界面上做不了。
其实 agent-desktop 那篇文末自己给了答案,只是被写进了「适用边界」:「我之前测滴答清单时碰到一排长得一模一样、没有名字的输入框,它在后台模式下直接拒绝修改,得允许它加 --headed 切到前台、用真鼠标去点。」
这句比「55 次 0 失误」重要得多。它说明不截图这条路线有一个硬边界,而且这个边界不在模型侧,在界面侧。
ref 定位,本质是拿序号代替坐标
不截图的 Agent 怎么定位:读操作系统的无障碍树,给每个控件编一个号,让模型输出「点 e17」。