Vercel Labs' skill: Teaches agents to use the agent-desktop command line to read and operate desktop applications on macOS, Windows, and Linux via the system accessibility tree—first taking a snapshot to obtain a JSON of UI elements with reference IDs, then clicking, typing, and checking based on those IDs, and finally re-snapshotting to verify, without relying on screenshots or pixel matching.
Security Audit: Not audited
- Not yet scanned
Rationale
- The loop is specifically defined: for small apps, directly use snapshot -i; for complex apps, first use --skeleton to get a shallow map, then use --root to drill into a specific area; references are in the format @s8f3k2p9:e1, valid only within the current snapshot, and if the UI changes, a STALE_REF error is reported, requiring a new snapshot
- Retry rules prevent duplicate submissions: retries are only allowed if disposition.retry is safe (i.e., not_delivered); in other cases, read the current UI state before deciding, to avoid double-clicking, double-submitting, or toggling undoing itself
- Headless by default: operates via accessibility interfaces, does not steal focus, move the mouse, or use the clipboard; --headed is added only if the user explicitly allows physical input, and switching to bypass headless failures is not permitted
- SKILL.md is very short; detailed command documentation is provided by the program itself (agent-desktop skills get desktop / platform) and aligned with the version; the README states Apache-2.0 license, and the author claims there are 60 command names in total, of which 56 are available
- skills.sh installed 536 times in 24 hours (indicating attention only)
Usage
- Installation commands will be displayed after the audit passes; first install the command line via npm install -g agent-desktop, and on first use, have the agent run agent-desktop status and agent-desktop skills get platform
- No account or cloud service required; on macOS, grant accessibility permissions to the terminal (agent-desktop permissions --guide); on Linux, a graphical session and at-spi2-core are required; on Windows, only applications with the same permission level as the terminal can be operated
Risks
- Not audited in this cycle, fixed at C-level, installation commands not displayed; SKILL.md instructs agents to execute agent-desktop commands to operate local applications, which constitutes command execution and aligns with its purpose; Snyk audit on skills.sh is Warn
- Agents can read and write the clipboard, take screenshots, launch and close applications, and read notifications; the README warns that snapshots and screenshots may contain private content and should not be placed in public logs; the optional JEV assistant sends screen descriptions to your configured model service
- On Linux, mouse input and window control primarily support X11; under Wayland, screenshots go through the system portal and clipboard requires wl-clipboard; this was not run in the current test, so operational reliability is unknown