Browser Automation with agent-browser

intent

agent-browser is a headless browser automation CLI for AI agents and developers. use it when you need to programmatically interact with websites: navigate pages, fill forms, click buttons, extract data, take screenshots, test web apps, handle authentication, or automate any browser-based workflow. triggers include explicit requests like "open a website", "fill out a form", "login to a site", "scrape data", or any task requiring programmatic web interaction. it persists a background browser daemon across command invocations, so chaining operations is efficient and state carries forward within a session.

inputs

required

• url (string): target webpage, supports http://, https://, and file:// (local files) protocols. file:// access requires --allow-file-access flag.
• agent-browser cli: installed via npm install -g agent-browser or npx agent-browser:*. verify with agent-browser --version.

optional external connections

• chrome/chromium instance (existing): use --auto-connect to discover a running Chrome with remote debugging enabled (--remote-debugging-port=9222), or specify explicit CDP port via --cdp <port>. useful for reusing existing browser context.
• proxy server (http/https/socks5): pass via --proxy <url> flag or AGENT_BROWSER_PROXY env var. required for geo-testing, rotating proxies, or routing through corporate proxies. proxy auth supported in URL format http://user:pass@host:port.
• ios simulator or real device (optional, macOS only): requires Xcode and Appium (npm install -g appium && appium driver install xcuitest). real devices need pre-configuration; use UDID from xcrun xctrace list devices.

environment variables

• AGENT_BROWSER_SESSION: default session name (overridden by --session flag).
• AGENT_BROWSER_CONFIG: path to custom config file (defaults to ~/.agent-browser/config.json then ./agent-browser.json).
• AGENT_BROWSER_ENCRYPTION_KEY: hex-encoded 32-byte key for encrypting saved session state at rest. generate via openssl rand -hex 32.
• AGENT_BROWSER_PROXY: proxy URL if not passed via flag.

configuration file

create agent-browser.json in project root for persistent settings (lowest priority) or ~/.agent-browser/config.json (system-wide). CLI flags override env vars which override config files. example:

{
  "headed": true,
  "proxy": "http://localhost:8080",
  "profile": "./browser-data",
  "timeout": 60000
}

procedure

1. initialize browser and navigate

input: url (http/https/file), optional session name, optional flags (headed, proxy, allow-file-access, auto-connect, cdp port) step: run agent-browser open <url> with desired flags. if using an existing session, prefix with --session <name>. background daemon spawns and loads page. output: browser loads page; stdout confirms navigation (e.g., "Navigated to https://example.com"). process waits for initial navigation, not full page load. edge case: slow pages may timeout at 60s (Playwright default). if timeout occurs, re-run and chain with explicit wait (see step 2).

2. wait for page readiness

input: optional wait mode (networkidle, domcontentloaded), optional selector, optional url pattern, optional millisecond duration, optional js condition. step: after open, run agent-browser wait --load networkidle (recommended for slow pages) or agent-browser wait <selector> to block until element exists, or agent-browser wait --url "**/pattern" for redirects, or agent-browser wait <ms> for fixed delay, or agent-browser wait --fn "condition" for custom js. chain with && to open if output not needed. output: command exits when condition met (0 exit code) or timeout (1 exit code after 60s default). edge case: networkidle mode waits for no pending network requests; can be slow on pages with background polling. domcontentloaded is faster but doesn't guarantee all resources loaded. use specific selectors when possible.

3. snapshot interactive elements and get refs

input: optional scope selector, optional flags (interactive-only -i recommended, include-cursor -C, json output -j). step: run agent-browser snapshot -i to list all clickable/fillable elements with auto-generated refs (@e1, @e2, etc.). output maps each ref to element type and text/label. if page has cursor-interactive divs (onclick handlers), add -C flag. to scope to subset, use -s "#selector". output: stdout lists elements in order: @e1 [input type="email"] placeholder="Email", @e2 [button] "Submit", etc. or json format with --json flag for parsing. edge case: dynamic content (lazy-loaded, ajax-driven) won't appear until loaded. re-snapshot after clicks/navigation. refs invalidate on page change; must re-snapshot to use fresh refs.

4. interact with elements (click, fill, select, check, scroll)

input: element ref from snapshot (@e1, etc.), optional interaction args (text for fill/type, option text for select, key for press, distance for scroll). step: run agent-browser click @e1 (click), agent-browser fill @e2 "text" (clear then type), agent-browser type @e2 "text" (type without clearing), agent-browser select @e3 "option text" (select dropdown), agent-browser check @e4 (check checkbox), agent-browser press <key> (press key like Enter, Tab, Escape), or agent-browser scroll <direction> <pixels> (scroll page). output: interaction executes; command exits 0 on success. element must be visible and clickable. edge case: if element is outside viewport, auto-scrolls into view. if element references stale (from old snapshot), interaction fails with error message "element not found". always re-snapshot after navigation or form submission.

5. capture visual information (screenshot, pdf, get text/url/title)

input: optional filename, optional flags (full-page --full, annotated --annotate, with element labels). step: run agent-browser screenshot <filename> to save png (or agent-browser screenshot --full for full-page capture). use --annotate to overlay numbered labels on interactive elements (each label maps to ref @eN). or agent-browser pdf <filename> for pdf output. or agent-browser get text @e1 to extract element text, agent-browser get text body > file.txt for all page text, agent-browser get url for current url, agent-browser get title for page title. output: file written to disk with relative or absolute path, or stdout for get commands. annotated screenshots cache refs for immediate use without re-snapshot. edge case: full-page screenshots may be very large on long pages. annotated mode useful for vision-based reasoning (models can see element positions and labels). refs from --annotate remain valid for subsequent interactions in same session.

6. re-snapshot after navigation or dynamic changes

input: same as step 3 (optional scope, flags). step: after any click that navigates, form submission, or dynamic load, run agent-browser snapshot -i again to refresh element refs. do not reuse old refs on new page. output: fresh element refs for new page state. edge case: refs are page-specific and browser session-specific. if browser restarted or session closed, all refs invalidate.

7. save and restore session state (authentication, cookies, localStorage)

input: optional session name, optional state filename, optional encryption key (env var or generated). step: after completing auth flow or setup, run agent-browser state save <filename> to persist cookies and localStorage. on next invocation, run agent-browser state load <filename> to restore. or use --session-name <name> flag on all commands; state auto-saves to ~/.agent-browser/sessions/<name>.json and auto-loads on next use. for encryption, set AGENT_BROWSER_ENCRYPTION_KEY env var before open/save. output: state file written (plain json or encrypted), loaded silently on next session. list saved states with agent-browser state list, show one with agent-browser state show <file>, clear with agent-browser state clear <name>, clean old states with agent-browser state clean --older-than <days>. edge case: state only persists cookies and localStorage, not javascript variables. encrypted state requires matching key on load; wrong key causes load failure. state files are session-specific; cannot reuse auth state across different user accounts or sites unless manually exported/imported.

8. advanced: chain commands and use parallel sessions

input: multiple commands, optional session names for isolation. step: chain operations with && when output of earlier command not needed (e.g., agent-browser open https://example.com && agent-browser wait --load networkidle && agent-browser snapshot -i). for concurrent work on multiple sites, use distinct session names: agent-browser --session site1 open https://a.com and agent-browser --session site2 open https://b.com in parallel. manage with agent-browser session list and agent-browser --session <name> close. output: chained commands execute sequentially in same shell invocation; background daemon persists browser. parallel sessions are isolated; daemon spawns separate browser instance per session. edge case: chaining is safe and more efficient than spawning separate shell invocations. if command output needed to decide next step (e.g., parse snapshot to find ref), run separately to capture stdout. avoid chaining if early command might fail (exit code will stop chain). cleanup: always close session to avoid leaked processes.

9. advanced: evaluate javascript and use semantic locators

input: javascript expression (single-line simple or multiline via --stdin or base64 via -b), or semantic locator pattern (text, role, label, placeholder, testid). step: run agent-browser eval 'expression' for simple js, or agent-browser eval --stdin <<'EVALEOF' ... EVALEOF for complex (multiline, nested quotes), or agent-browser eval -b "$(echo -n 'expression' | base64)" for generated scripts. or use agent-browser find text "Label" click to click element containing text, agent-browser find role button click --name "Submit" to click button by name, agent-browser find label "Email" fill "text" to fill by label, etc. output: js eval returns result as json-serialized output. find commands resolve to element and execute action. both bypass snapshot refs (useful when refs stale or snapshot missed element). edge case: shell quoting can corrupt js expressions (especially !, $(), backticks, nested quotes). use --stdin with heredoc or -b base64 to avoid shell interpretation. find commands are slower than ref-based interaction (they re-query on each call).

10. close session and cleanup

input: optional session name. step: run agent-browser close (closes default session) or agent-browser --session <name> close (closes specific session). cleans up browser process and saves session state if using --session-name. output: browser closed, daemon process exits or remains for other sessions. edge case: if session not closed (e.g., script interrupted), daemon may still hold resources. manually kill with agent-browser close next time, or check agent-browser session list to see orphaned sessions.

decision points

if page loads slowly or consistently times out

• decision: is the slow load due to initial page load or dynamic content after page load?
  • if initial page load: use agent-browser wait --load networkidle after open to block until network settles. or use agent-browser wait --fn "document.readyState === 'complete'" for javascript readiness.
  • if dynamic content (ajax, dropdowns, lazy-load): snapshot first, then wait for specific element with agent-browser wait @e1 or agent-browser wait "#selector" before interacting.

if element ref is not found or interaction fails

• decision: was the page state changed since last snapshot?
  • if yes (page navigated, form submitted, modal opened): always re-snapshot with agent-browser snapshot -i and use fresh refs.
  • if no: element may be outside viewport (auto-scroll handles this). if still fails, try agent-browser find semantic locators instead of refs, or take annotated screenshot to verify element exists and is labeled.

if using authentication (login, oauth, 2fa)

• decision: should state be saved for reuse?
  • if yes (same user, same session): use agent-browser state save auth.json after login, then agent-browser state load auth.json before subsequent operations.
  • if yes (persistent across restarts): use --session-name <name> flag on all commands; state auto-persists and auto-loads.
  • if no (single-use, ephemeral): skip save; state lost on close.
  • if 2fa required: snapshot after auth step to detect 2fa prompt, interact with it, then save state.

if running multiple agents or tasks concurrently

• decision: should sessions be isolated?
  • if yes (separate sites, accounts, or state): use --session agent1, --session agent2, etc. on all commands. each agent gets isolated browser and state.
  • if no (single agent, sequential tasks): use default session (no --session flag). reuse same browser across tasks.

if needing to interact with local files (pdf, html)

• decision: can file:// protocol be used?
  • if yes (local machine, no security concerns): use --allow-file-access flag and agent-browser open file:///path/to/file.pdf.
  • if no (security sandboxing required): use http server to serve files, open via http://localhost:port/file.

if targeting mobile or ios

• decision: is ios simulator/device available?
  • if yes (macOS with Xcode and Appium): use agent-browser -p ios open <url>. snapshot, interact, and screenshot work same as desktop. use agent-browser -p ios tap @e1 for tap (click alias) and agent-browser -p ios swipe <direction> for gestures.
  • if no: use desktop browser and test via responsive design mode (user agent strings) or skip mobile testing.

if javascript evaluation or complex element selection needed

• decision: is a simple snapshot ref sufficient?
  • if yes: use agent-browser click @e1 (fastest, cached).
  • if no (complex logic, dynamic refs, canvas elements): use agent-browser eval --stdin to run javascript and parse results, or agent-browser find semantic locators to locate by text/role/placeholder.

output contract

successful navigation

• format: stdout confirms Navigated to <url>. exit code 0.
• side effect: background browser daemon running, page loaded (initial html, not necessarily full rendering).

successful snapshot

• format: stdout lists elements line-by-line: @e1 [input type="email"], @e2 [button] "Submit". or json with --json flag: [{"ref":"@e1","tag":"input","type":"email",...}]. exit code 0.
• lifetime: refs valid only for current page state and session. invalidate on navigation, form submission, or dom changes.

successful interaction (click, fill, select, check)

• format: stdout minimal or empty. exit code 0. element state changes (checked, filled, focused).
• side effect: page may navigate, reload, or fire dom events (ajax requests, validation). always re-snapshot if refs may be invalidated.

successful capture (screenshot, pdf)

• format: file written to disk. stdout prints filename/path. exit code 0.
• lifetime: files persist on disk until manually deleted.

successful get (text, url, title)

• format: stdout returns value (text content, url string, title string). exit code 0. or json with --json flag.

successful state save

• format: file written to disk at specified path or ~/.agent-browser/sessions/<name>.json. stdout confirms path. exit code 0. file is json (plain or encrypted if key set).

successful state load

• format: no stdout (silent). exit code 0. cookies and localStorage restored to browser. session ready for use.

session list

• format: stdout lists active sessions: site1, site2 (one per line). exit code 0.

failed command

• format: stderr contains error message (element not found, timeout, invalid selector, auth failure, network error). exit code 1.
• no side effects: failed interaction does not modify page state. safe to retry.

outcome signal

user knows skill worked when:

1. page loaded: url matches expectation and content visible (if headed flag set, can see in window). stdout says "Navigated to ".

2. elements found and labeled: snapshot output lists elements with refs. if using --annotate screenshot, image shows numbered labels on interactive elements matching output refs.

3. interaction succeeded: element state changed (input filled, button clicked, checkbox checked, page navigated). next snapshot shows updated page state or new page loaded.

4. form submitted: after click on submit button, page navigates or shows confirmation. re-snapshot confirms you're on success page (check url or new elements).

5. login successful: after filling credentials and clicking login, page navigates to authenticated area (dashboard, account page, etc.). state file saved for reuse.

6. data extracted: get text returns content. screenshot or pdf files written to disk. eval returns parsed json result.

7. session persisted: state save writes file. state load restores without requiring re-login. next session with same --session-name auto-loads state.