agent-desktop 1.2.0 adds MCP support for background desktop control
Vercel Labs released agent-desktop 1.2.0 with MCP support, according to 掘金 · 人工智能. The tool reads the operating system accessibility tree and assigns numbered UI controls so an AI can trigger them without screenshots or taking over the mouse and keyboard. In the author's WorkBuddy test, it made 55 error-free Calculator clicks and recorded a total of 1746 in Notepad, although actions were slow, consumed credits per step, and could not handle every interface in the background.
The release offers a concrete MCP-based approach to desktop automation that avoids screenshot-based coordinate clicking, but the available evidence comes from one promotional-leaning hands-on report.
以前让 AI 操作电脑上的软件,常规做法都是屏幕截图塞给模型看、猜像素坐标再用鼠标点过去:既费 token,干活时还会直接抢占你的鼠标和键盘。
Vercel Labs 昨天(11 月 5 日)给开源工具 agent-desktop 发布了 1.2.0 版本,加入了 MCP 支持。它的原理是直接读取操作系统的无障碍树,给界面的每个控件编号,让 AI 依据编号精准触发;操作默认在后台执行,全程不需要截图,也不会抢占你的鼠标键盘。
我一年光 AI 订阅就要花费 1.7 万元,月底对账非常繁琐,正好拿它做测试:将其接入 WorkBuddy,让它控制系统计算器把 15 笔账单逐笔加总、填入记事本,实测全程 0 截图,55 次按键没有一次点错。
