What problem it solves

Coding agents need a consistent way to inspect, control, debug, and verify applications across mobile, web, TV, and desktop targets. They also need repeatable workflows and reviewable evidence across local devices, simulators, CI, and remote device clouds.

Key features

• Inspect and control app interfaces — Agents can read accessibility snapshots, act through refs and selectors, and perform interactions such as pressing, filling, scrolling, gesturing, waiting, and asserting state.

  Source: README

• Capture diagnostic evidence — The project provides screenshots, video, logs, traces, network data, performance samples, crash details, and React profiles for diagnosing failures and reviewing changes.

  Source: README

• Use CLI, MCP, or Node.js interfaces — The same runtime is available through a command-line interface, a built-in MCP server, and a typed Node.js API for agents and orchestration code.

  Source: README

• Record and replay workflows — Working steps can be saved as `.ad` scripts for local use or CI, with the option to export strict Maestro YAML.

  Source: README

• Coordinate local and remote devices — Sessions preserve device state, coordinate ownership across parallel agent worktrees, and support local, CI/CD, cloud, and remote-device workflows.

  Source: README

Quick start

Copied verbatim from the project's README.

npm install -g agent-device@latest
agent-device doctor
agent-device help workflow

# Start a session.
agent-device open Contacts --platform ios

# Inspect the screen. The example below shows the output; refs vary.
agent-device snapshot -i
# @e2 [button] "Add"

# Use the ref and wait for the UI to settle.
agent-device press @e2 --settle
# The diff includes:
# + @e7 [text-field] "First name"

agent-device fill @e7 "Ada" --settle
# The next diff shows changed values and current refs:
# - @e7 [text-field] "First name"
# + @e14 [text-field] "Ada"
# = @e15 [text-field] "Last name"

# Capture evidence and close the session.
agent-device screenshot ./contact-form.png
agent-device close

import { createAgentDeviceClient } from 'agent-device';

const client = createAgentDeviceClient({ session: 'qa-run' });
try {
  await client.apps.open({ app: 'com.apple.Preferences', platform: 'ios' });
  const snapshot = await client.capture.snapshot({ interactiveOnly: true });
  const button = snapshot.nodes.find((node) => node.role === 'button');
  if (button) await client.interactions.press({ ref: button.ref });
} finally {
  await client.sessions.close();
}

Who it may suit

This may suit developers and teams building AI coding agents, agent-driven QA workflows, or custom agent runtimes that need to operate applications across devices and simulators.

Topics to discuss

• Which accessibility-first interaction patterns are most reliable across supported platforms?

• How should agents balance accessibility snapshots with screenshots and video evidence?

• What workflow should teams standardize for recording, replaying, and reviewing device sessions in CI?

Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.

Read the original README https://github.com/callstack/agent-device#readme

ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.