What problem it solves

AI agents need a way to observe and operate desktop applications through their native user interfaces. agent-desktop provides this interaction model by reading the accessibility tree, returning actionable references in JSON, and allowing agents to verify the result by reading the interface again.

Key features

• Accessibility-tree snapshots with actionable references — Snapshot desktop applications and receive their accessibility tree as JSON. Every element an agent can act on is identified by a reference.

  Source: README

• Observe, act, and verify interaction loop — Agents can inspect the interface, act on a reference, and take another snapshot to verify the result. Stale or ambiguous references are reported so the agent can refresh its view.

  Source: README

• CLI, library, and MCP interfaces — The same core is available as a command-line application and an in-process C library. It can also expose every command as an MCP tool over stdio.

  Source: README

• Cross-platform desktop support — Platform adapters use the native accessibility API on macOS, Windows, and Linux. The README documents supported operating-system versions, architectures, and display environments for each platform.

  Source: README, crates

Quick start

Copied verbatim from the project's README.

npm install -g agent-desktop
agent-desktop status                # platform, permissions, supported surfaces
agent-desktop permissions --guide   # macOS: walks you through the Accessibility grant

agent-desktop snapshot --app Finder -i      # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8            # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i      # observe again to verify

Who it may suit

This may suit developers building AI agents that need to observe or operate desktop applications across macOS, Windows, or Linux. It may also suit teams integrating desktop automation through a CLI, C library, or MCP server.

Topics to discuss

• Which agent workflows would benefit most from accessibility-tree references instead of screenshot-based interaction?

• How should agents balance headless actions with headed physical input across different operating systems?

• Which commands or integration patterns should receive more examples and troubleshooting guidance?

Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.

Read the original README https://github.com/vercel-labs/agent-desktop#readme

ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.