agent-desktop

CLI tool enabling AI agents to observe and control desktop applications via native OS accessibility trees.

core principle: agent-desktop is NOT an AI agent. it is a tool that AI agents invoke. it outputs structured JSON with ref-based element identifiers. the observation-action loop lives in the calling agent.

intent

use agent-desktop when you need to automate desktop GUI interactions without writing platform-specific code. the skill lets you click buttons, fill forms, navigate menus, read UI state, toggle checkboxes, scroll, drag, type text, take screenshots, manage windows, and access clipboard across 54 commands. triggers on: "click button", "fill form", "open app", "read UI", "automate desktop", "accessibility tree", "snapshot app", "type into field", "navigate menu", "toggle checkbox", "take screenshot", or any desktop GUI interaction task. currently supports macOS 12+ with Windows and Linux planned.

inputs

installation:

npm install -g agent-desktop
# or
bun install -g --trust agent-desktop

system requirements:

• macOS 12+ (Phase 1 only; Windows and Linux support planned)
• accessibility permission granted to your terminal application
• running GUI application to automate (launch via agent-desktop launch if needed)

environment setup:

• no API keys or external auth required
• terminal must have Accessibility permission: System Settings > Privacy > Accessibility > add your terminal app
• check permission with: agent-desktop permissions
• request permission with: agent-desktop permissions --request

reference documentation (external):

• references/commands-observation.md: snapshot, find, get, is, screenshot, list-surfaces with all flags and examples
• references/commands-interaction.md: click, type, set-value, select, toggle, scroll, drag, keyboard, mouse command selection
• references/commands-system.md: launch, close, windows, clipboard, wait, batch, status, permissions, version
• references/workflows.md: 12 common patterns (forms, menus, dialogs, scroll-find, drag-drop, async wait, anti-patterns)
• references/macos.md: macOS permissions/TCC, AX API internals, smart activation chain, surfaces, Notification Center, troubleshooting

procedure

every automation follows the observe-act loop:

1. observe: run agent-desktop snapshot --app "App Name" -i to fetch the current accessibility tree with refs for interactive elements only. parse the JSON output to locate target elements by ref (@e1, @e2, etc.). store the refmap in memory; refs are scoped to this snapshot and become stale after UI changes.

2. reason: examine the JSON structure, identify the interactive element (button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell) matching your intent. note its ref identifier.

3. act: invoke the appropriate interaction command (click, type, select, toggle, scroll, drag, etc.) with the target ref. example: agent-desktop click @e5 or agent-desktop type @e2 "hello". if the action requires coordinates instead, use mouse-click --xy X,Y.

4. verify: run agent-desktop snapshot --app "App Name" -i again to confirm the UI state changed as expected. parse the new refmap and continue if more actions are needed.

5. repeat: loop through steps 1-4 until the task is complete (form submitted, menu closed, file saved, etc.).

ref system:

• refs are assigned depth-first in snapshot output: @e1, @e2, @e3, etc.
• only interactive elements (button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell) receive refs.
• static text, groups, and containers remain in the tree for context but have no ref.
• refs are deterministic within a snapshot but NOT stable across snapshots if the UI changed. after any action that modifies the DOM or view hierarchy, re-run snapshot for fresh refs.

command groups (54 total):

observation (6 commands):

agent-desktop snapshot --app "App" -i           # accessibility tree with refs
agent-desktop screenshot --app "App" out.png    # PNG screenshot
agent-desktop find --app "App" --role button    # search by role/name
agent-desktop get @e1 --property text           # read element property
agent-desktop is @e1 --property enabled         # check element state
agent-desktop list-surfaces --app "App"         # available surfaces (menu, tooltip, etc.)

interaction (17 commands):

agent-desktop click @e5                         # click element
agent-desktop double-click @e3                  # double-click
agent-desktop triple-click @e2                  # triple-click (select line)
agent-desktop right-click @e5                   # right-click (context menu)
agent-desktop type @e2 "hello"                  # type text into focused element
agent-desktop set-value @e2 "new value"         # set value directly (bypass focus)
agent-desktop clear @e2                         # clear element value
agent-desktop focus @e2                         # set keyboard focus
agent-desktop select @e4 "Option B"             # select dropdown option by label
agent-desktop toggle @e6                        # toggle checkbox or switch
agent-desktop check @e6                         # idempotent check (enable)
agent-desktop uncheck @e6                       # idempotent uncheck (disable)
agent-desktop expand @e7                        # expand disclosure/collapse group
agent-desktop collapse @e7                      # collapse disclosure/collapse group
agent-desktop scroll @e1 --direction down       # scroll element (up/down/left/right)
agent-desktop scroll-to @e8                     # scroll element into view
agent-desktop press return --app "App"          # send key or combo to app

keyboard & mouse (12 commands):

agent-desktop press cmd+c                       # key combo (cmd, ctrl, shift, option)
agent-desktop key-down shift                    # press and hold key
agent-desktop key-up shift                      # release held key
agent-desktop hover @e5                         # move cursor to element
agent-desktop hover --xy 500,300                # move cursor to coordinates
agent-desktop drag --from @e1 --to @e5          # drag between elements
agent-desktop mouse-click --xy 500,300          # click at coordinates
agent-desktop mouse-move --xy 100,200           # move cursor to coordinates
agent-desktop mouse-down --xy 100,200           # press mouse button
agent-desktop mouse-up --xy 300,400             # release mouse button
agent-desktop mouse-scroll --xy 500,300 --direction down  # scroll at coordinates
agent-desktop mouse-scroll-momentum --xy 500,300 --direction down --amount 5  # momentum scroll

app & window (12 commands):

agent-desktop launch "System Settings"          # launch app and wait for startup
agent-desktop close-app "TextEdit"              # quit gracefully
agent-desktop close-app "TextEdit" --force      # force kill (SIGKILL)
agent-desktop list-windows --app "Finder"       # list open windows
agent-desktop list-apps                         # list all running GUI apps
agent-desktop focus-window --app "Finder"       # bring window to front
agent-desktop resize-window --app "App" --width 800 --height 600  # resize
agent-desktop move-window --app "App" --x 0 --y 0  # reposition
agent-desktop minimize --app "App"              # minimize window
agent-desktop maximize --app "App"              # maximize window
agent-desktop restore --app "App"               # restore from minimized/maximized
agent-desktop activate-window --app "App" --title "Partial Title"  # activate by app and optional title

notifications (6 commands):

agent-desktop list-notifications                # list all system notifications
agent-desktop list-notifications --app "Slack"  # filter by app
agent-desktop list-notifications --text "deploy" --limit 5  # filter by text content
agent-desktop dismiss-notification 1            # dismiss by index
agent-desktop dismiss-all-notifications         # dismiss all
agent-desktop notification-action 1 --action "Reply"  # click action button on notification

clipboard (3 commands):

agent-desktop clipboard-get                     # read clipboard contents
agent-desktop clipboard-set "text"              # write to clipboard
agent-desktop clipboard-clear                   # clear clipboard

wait (6 commands):

agent-desktop wait 1000                         # pause (ms)
agent-desktop wait --element @e5 --timeout 5000 # wait for element to exist
agent-desktop wait --window "Title"             # wait for window to open
agent-desktop wait --text "Done" --app "App"    # wait for text in accessibility tree
agent-desktop wait --menu --app "App"           # wait for context menu to open
agent-desktop wait --menu-closed --app "App"    # wait for menu to dismiss

system (5 commands):

agent-desktop status                            # health check
agent-desktop permissions                       # check accessibility permission
agent-desktop permissions --request             # trigger permission dialog
agent-desktop version --json                    # version info
agent-desktop batch '[...]' --stop-on-error     # batch multiple commands

decision points

permission flow:

• if accessibility permission is not granted, agent-desktop status returns error code PERM_DENIED.
  • do this: run agent-desktop permissions --request to trigger the system dialog, or direct user to System Settings > Privacy > Accessibility > add terminal app.
  • else: all commands fail with PERM_DENIED until permission is granted.

app not running:

• if the target app is not running, snapshot --app "App Name" returns error code APP_NOT_FOUND.
  • do this: run agent-desktop launch "App Name" first, then retry snapshot.
  • else: skip launch and proceed if the app must already be open (e.g., automating an always-running service like Slack).

element not found (stale ref or wrong ref):

• if a ref does not exist in the current refmap, action commands return error code ELEMENT_NOT_FOUND.
  • do this: immediately re-run snapshot --app "App" to fetch fresh refs, then re-issue the action with the new ref.
  • else: the action is silently skipped or hangs waiting for the element (depending on command).

action not supported by element:

• if an element does not support the requested action (e.g., toggle on a read-only checkbox), the command returns error code ACTION_NOT_SUPPORTED.
  • do this: try an alternative approach (e.g., click instead of toggle, or use coordinate-based mouse-click).
  • else: the command fails and the task stalls.

timeout during wait:

• if a wait command does not meet its condition within the --timeout window (default 5000ms), it returns error code TIMEOUT.
  • do this: increase --timeout for slow apps or network delays, or verify the expected state with snapshot to see if the condition is actually met.
  • else: the automation advances without the expected state change, leading to wrong element refs or missing actions.

async/delayed UI changes:

• after launching an app, opening a dialog, or triggering a dropdown, the UI may not update immediately.
  • do this: use agent-desktop wait --element @e5 --timeout 5000 or wait --text "Label" to block until the new state exists, then snapshot again.
  • else: the next snapshot may capture partial or stale UI, causing ref mismatches.

ref stability across snapshots:

• refs are snapshot-scoped and become stale if the UI changes (elements added, removed, reordered).
  • do this: always re-run snapshot after any action that modifies the DOM (click, type, toggle, scroll, etc.) before using old refs.
  • else: stale refs cause STALE_REF errors on subsequent actions.

rate limits and performance:

• agent-desktop has no built-in rate limiting, but macOS accessibility API can be slow under heavy load.
  • do this: batch related commands using agent-desktop batch to reduce per-command overhead; add deliberate waits between rapid actions; profile with time to detect bottlenecks.
  • else: commands may timeout or produce inconsistent results if the AX API is saturated.

network and file dependencies:

• agent-desktop does not require network access, but screenshots and other commands may write files to disk.
  • do this: ensure the output path is writable and has sufficient disk space (e.g., agent-desktop screenshot --app "App" /tmp/out.png).
  • else: file operations fail with OS-level permission errors.

terminal accessibility permission expiry:

• on macOS, accessibility permissions are stored in the TCC.db database and are per-app per-user.
  • do this: if you upgrade the terminal app, reinstall it, or change its code signature, re-run agent-desktop permissions --request to re-grant access.
  • else: all commands fail with PERM_DENIED until permission is re-granted.

coordinate-based fallback:

• if element-based refs are unreliable (e.g., for custom UI frameworks or broken accessibility trees), fallback to coordinate-based commands.
  • do this: use agent-desktop screenshot --app "App" to get coordinates, then use mouse-click --xy X,Y, hover --xy X,Y, or drag --from --to with coordinates.
  • else: the automation is brittle and breaks if the UI layout changes.

output contract

every command returns JSON on stdout following this envelope:

success:

{
  "version": "1.0",
  "ok": true,
  "command": "snapshot",
  "data": { ... }
}

error:

{
  "version": "1.0",
  "ok": false,
  "command": "click",
  "error": {
    "code": "STALE_REF",
    "message": "ref @e5 not found in current refmap",
    "suggestion": "re-run snapshot to fetch fresh refs"
  }
}

exit codes:

• 0: success (ok=true)
• 1: structured error (ok=false with error.code)
• 2: argument/syntax error (invalid command or flags)

snapshot data format (observation commands):

• data.tree: accessibility tree as nested JSON. each node has role, label (optional), children (array), ref (string, only for interactive elements).
• data.refmap: flat dict mapping ref to element info, e.g. {"@e1": {"role": "button", "label": "OK", "enabled": true}, "@e2": {"role": "textfield", "label": "Username", "value": ""}}.
• data.app: app bundle name (e.g., "com.apple.finder").
• data.window: window title if applicable.

screenshot data format:

• data.path: file path where PNG was written (absolute path).
• data.size: { "width": number, "height": number } in pixels.

find data format:

• data.matches: array of matching elements, each with ref, role, label, enabled, position (x, y, width, height).

get/is data format:

• data.value: the property value (string, boolean, number, or null).
• data.property: the property name that was retrieved.

list-windows data format:

• data.windows: array of window objects, each with title (string), position (x, y), size (width, height), focused (boolean).

list-apps data format:

• data.apps: array of app names (bundle identifiers or display names).

list-notifications data format:

• data.notifications: array of notification objects, each with index (int), app (string), title (string), body (string), actions (array of action strings).

clipboard data format:

• data.text: clipboard contents as string (or null if empty).

wait data format:

• data.elapsed: milliseconds waited before condition was met.
• data.condition: the wait condition that was satisfied (e.g., "element @e5 exists", "text 'Done' found").

error codes and recovery:

code	meaning	recovery
PERM_DENIED	accessibility permission not granted	run agent-desktop permissions --request or add terminal to System Settings > Privacy > Accessibility
ELEMENT_NOT_FOUND	ref not in current refmap	re-run snapshot, use fresh ref
APP_NOT_FOUND	app not running	launch app with agent-desktop launch "App Name"
ACTION_FAILED	AX action rejected by OS	try alternative approach or coordinate-based command (mouse-click --xy)
ACTION_NOT_SUPPORTED	element cannot perform this action	use different command (e.g., click instead of toggle)
STALE_REF	ref from old snapshot, UI has changed	re-run snapshot immediately
WINDOW_NOT_FOUND	no window matching app name or title	check app name with list-windows or list-apps
TIMEOUT	wait condition not met within timeout window	increase --timeout value or verify condition with snapshot
INVALID_ARGS	bad command arguments or syntax	check command flags and argument order in reference docs
RATE_LIMITED	AX API overloaded	add delay with wait or batch related commands
FILE_IO_ERROR	screenshot or file write failed	ensure output directory is writable and has disk space

outcome signal

you know the skill worked when:

1. snapshot returns a non-empty refmap: parse data.refmap and confirm refs exist for target elements. if refmap is empty, the accessibility tree is broken or the app does not expose UI.