A desktop computer operation tool for AI agents: It reads the real interface structure of any application via the system accessibility tree, returns a JSON snapshot, and assigns a reference to each actionable element. The agent clicks or inputs based on these references and then takes another snapshot to verify, without relying on pixel recognition.
Stars 1833 (+22 / 24h)
Reasons
- References are bound to their respective snapshots. If the interface changes, it returns STALE_REF or AMBIGUOUS_TARGET and requires a new snapshot, preventing clicks on deleted rows. It operates in headless mode by default via the accessibility interface, without moving the mouse or stealing focus. Add --headed when real input is needed.
- 60 command names, 56 available: observation, actions on references, keyboard and mouse, application and window management, notifications, clipboard, waiting, and session tracking. For dense interfaces, first use snapshot --skeleton to get the skeleton, then drill down into a specific area.
- Each command outputs a unified JSON envelope. Errors include a code, whether it is safe to retry, and suggestions. A C library is also provided, allowing programs to call it in-process.
Usage
- npm install -g agent-desktop; macOS requires accessibility permissions (agent-desktop permissions --guide), and Linux requires a graphical session and at-spi2-core.
- Have the agent read the built-in documentation first: agent-desktop skills get desktop and skills get platform; or npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop.
- Minimal loop: agent-desktop snapshot --app Finder -i → agent-desktop click <reference from snapshot> → take another snapshot to verify.
Risks
- Snapshots, clipboard, screenshots, and logs may contain private content from the desktop. The author warns against including them in public logs. The optional JEV assistance sends screen descriptions to your configured model provider.
- Linux capabilities depend on X11 / Wayland and GNOME, with some features missing. Official Homebrew and Scoop packages are still being submitted. License: Apache-2.0.