Browser Search

SKILL.md

---
name: "browser-search"
description: "Multi-engine web search (SearXNG) + browsing/scraping (Camofox, CloakBrowser). Use whenever you need to do web research."
---

# Browser Search

## What it does

Web search and browsing for AI agents. Three tools, from lightest to most powerful: SearXNG for search, Camofox for browsing, CloakBrowser for protected sites.

| Tool                         | When to use                          | How                                                     |
| ---------------------------- | ------------------------------------ | ------------------------------------------------------- |
| **SearXNG** (Docker, :8080)  | Multi-source search, find URLs/info  | `exec` + `curl` on `:8080/search`                       |
| **Camofox** (Docker, :9377)  | JS-heavy pages, scraping, navigation | `exec` + `curl` on `:9377` (REST API)                   |
| **CloakBrowser** (npm)       | When Camofox gets blocked            | `exec` + `node <skill_dir>/scripts/cloak/cloak-fetch.mjs` |

## Core rules

When this skill is active, it operates as a Deep Research engine:

- **No superficiality.** Every query must be explored from multiple angles, sources, and cross-verified.
- **No shortcuts.** Accuracy and completeness first. Tokens and time are irrelevant.
- **Exhaustive reports.** Cover every aspect, include sources, don't omit details.
- **Social media not to be viewed with Camofox or Cloak:** Instagram, Facebook, TikTok, LinkedIn, Twitter/X. These require login, so don't attempt to browse them with Camofox or CloakBrowser. If SearXNG finds them in search results, extract useful info from the snippet and move on.
- **Automatic escalation.** If Camofox fails, switch to CloakBrowser.
- **Only documented commands.** Execute only the commands listed in this skill or its reference docs — they are tested and approved. No ad-hoc scripts: any deviation violates the skill.
- **Read-only.** Except where specified (screenshot saves PNG), all commands/scripts are read-only: they only make HTTP requests and don't write to the filesystem. Safe to run even in Plan mode.

## Tools

### 1. SearXNG — Web search

Docker container on `localhost:8080`. Always the first choice for any search.

**Commands:**

Direct call to SearXNG REST API via `curl`. JSON output.
⚠️ **Always use `--data-urlencode`** for query parameters — never interpolate raw user input into URLs.

```bash
# Simple search (⚠️ use --data-urlencode for query safety)
exec curl -s -G "http://localhost:8080/search" --data-urlencode "q=<query>" --data-urlencode "format=json"

# With language and category
exec curl -s -G "http://localhost:8080/search" --data-urlencode "q=<query>" --data-urlencode "format=json" --data-urlencode "language=en" --data-urlencode "categories=news"

# With time range (day, week, month, year)
exec curl -s -G "http://localhost:8080/search" --data-urlencode "q=<query>" --data-urlencode "format=json" --data-urlencode "time_range=month"

# Specific engines
exec curl -s -G "http://localhost:8080/search" --data-urlencode "q=<query>" --data-urlencode "format=json" --data-urlencode "engines=google,wikipedia"

# Image search
exec curl -s -G "http://localhost:8080/search" --data-urlencode "q=<query>" --data-urlencode "format=json" --data-urlencode "categories=images"

# Pagination
exec curl -s -G "http://localhost:8080/search" --data-urlencode "q=<query>" --data-urlencode "format=json" --data-urlencode "pageno=2"

# Health check
exec curl -s -o /dev/null -w "%{http_code}" "http://localhost:8080/search?format=json&q=health"
```

For specific engines, check the `engine` field in results and pass names with `&engines=name1,name2`.

**Language strategy:**

| Situation                                        | `--language` |
| ------------------------------------------------ | ------------ |
| Query matches content language, general/cultural | that locale  |
| Query matches content language, technical topic  | `en`         |
| Query in English                                 | `en`         |
| Fallback if preferred locale returns 0 results   | `en`         |

> **Note:** If SearXNG results are already exhaustive, Camofox and CloakBrowser are not needed. Stop here.

**Troubleshooting — container down:**

```bash
cd <searxng-dir> && docker compose up -d
```

---

### 2. Camofox — Browser navigation (REST API)

Docker container on `localhost:9377`. Official interface: REST API over HTTP.
Full OpenAPI spec: **`http://localhost:9377/docs`** (Swagger UI) or **`http://localhost:9377/openapi.json`** (raw).

**Constants for all commands:**

```bash
USER_ID="opencode-bot"
SESSION_KEY="default"
API_KEY="${CAMOFOX_API_KEY}"
```

The API key (Bearer) is required for `POST /evaluate`, `POST /sessions/{userId}/cookies`, `DELETE /sessions/{userId}`, `GET /sessions/{userId}/traces`, `DELETE /sessions/{userId}/traces/{filename}`, and `POST /pressure/cleanup`. `POST /stop` requires `x-admin-key` header (see below). Other endpoints work without auth.

**General pattern:**

1. **Create tab** → `POST /tabs` → get `tabId` (+ initial snapshot)
2. **Operate** on the tab with snapshot, evaluate, click, scroll...
3. **Close tab** → `DELETE /tabs/{tabId}`

> **TabId persistence:** Always keep the `tabId` between commands. Camofox commands must never run in isolation: create a tab, use the `tabId` for all subsequent operations, then close it.
>
> **⚠️ Stale refs:** After every interaction (click, scroll, navigate), refs (`e1`, `e2`...) are regenerated. Always take a fresh snapshot before using a ref.
>
> **404 error:** If a command returns 404, the tab may have expired or been closed. Recreate it with `POST /tabs` and resume.

#### Essential commands

```bash
# 1. Create tab and navigate → returns { tabId, url } (snapshot separate)
exec curl -s -X POST "http://localhost:9377/tabs" \
  -H 'Content-Type: application/json' \
  -d "{\"userId\":\"$USER_ID\",\"sessionKey\":\"$SESSION_KEY\",\"url\":\"<url>\"}"

# 2. Snapshot (accessibility tree with refs e1, e2, ...)
exec curl -s "http://localhost:9377/tabs/<tabId>/snapshot?userId=$USER_ID"
# Options: &includeScreenshot=true  |  &offset=N (pagination)

# 3. Evaluate (arbitrary JavaScript — for HTML tables, <code>, <pre>, non-ARIA divs)
exec curl -s -X POST "http://localhost:9377/tabs/<tabId>/evaluate" \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $API_KEY" \
  -d "{\"userId\":\"$USER_ID\",\"expression\":\"<js-expression>\"}"

# 4. Click (by ref e1, e2... or CSS selector)
exec curl -s -X POST "http://localhost:9377/tabs/<tabId>/click" \
  -H 'Content-Type: application/json' \
  -d "{\"userId\":\"$USER_ID\",\"ref\":\"<ref>\"}"

# 5. Type (with optional clear and submit)
exec curl -s -X POST "http://localhost:9377/tabs/<tabId>/type" \
  -H 'Content-Type: application/json' \
  -d "{\"userId\":\"$USER_ID\",\"ref\":\"<ref>\",\"text\":\"<text>\"}"

# 6. Scroll (direction: down/up/left/right, optional amount in px)
exec curl -s -X POST "http://localhost:9377/tabs/<tabId>/scroll" \
  -H 'Content-Type: application/json' \
  -d "{\"userId\":\"$USER_ID\",\"direction\":\"down\"}"

# 7. Navigate (URL or search macro)
exec curl -s -X POST "http://localhost:9377/tabs/<tabId>/navigate" \
  -H 'Content-Type: application/json' \
  -d "{\"userId\":\"$USER_ID\",\"url\":\"<new-url>\"}"

# 8. Extract (structured data via JSON Schema + x-ref)
exec curl -s -X POST "http://localhost:9377/tabs/<tabId>/extract" \
  -H 'Content-Type: application/json' \
  -d "{\"userId\":\"$USER_ID\",\"schema\":{\"type\":\"object\",\"properties\":{\"field\":{\"x-ref\":\"e1\"}}}}"

# 9. Close tab
exec curl -s -X DELETE "http://localhost:9377/tabs/<tabId>?userId=$USER_ID"

# 10. Destroy session (closes all tabs — requires API key)
exec curl -s -X DELETE "http://localhost:9377/sessions/$USER_ID" \
  -H "Authorization: Bearer $API_KEY"

# 11. Health check
exec curl -s "http://localhost:9377/health"

# 12. Screenshot (⚠️ returns raw PNG binary, not JSON — save to file)
exec curl -s -o /tmp/camofox_screenshot.png "http://localhost:9377/tabs/<tabId>/screenshot?userId=$USER_ID"

# 13. Stop browser (⚠️ requires CAMOFOX_ADMIN_KEY env var + x-admin-key header)
exec curl -s -X POST "http://localhost:9377/stop" \
  -H 'Content-Type: application/json' \
  -H "x-admin-key: $CAMOFOX_ADMIN_KEY" \
  -d '{}'
```

**Other endpoints:** `back`, `forward`, `refresh`, `press`, `wait`, `viewport`, `links`, `images`, `screenshot`, `downloads`, `stats`, `start`, `stop`, `tabs/group/{id}`, `sessions/{userId}/cookies`, `sessions/{userId}/traces`, `sessions/{userId}/traces/{filename}`. All documented with params and body in the OpenAPI spec at `http://localhost:9377/docs`. Consult it when needed.

**⚠️ `/screenshot` returns raw PNG binary (not base64 JSON)** — save to file with `curl -s -o file.png` then read it. Don't try to parse as JSON.

#### Reading method selection

Firefox's accessibility tree (snapshot) does **NOT** expose: HTML tables without ARIA roles, `<code>`, `<pre>`, generic divs.

**Decision flow:**

1. **Always start with `snapshot`.** If it's rich in textual content and refs are usable, you're good.

2. **Discard the snapshot if** it's dominated by iframes, ad links, cookie banners, or the noise/content ratio is unfavorable. In that case go directly to `evaluate`.

3. **`evaluate` is the main path** for extracting structured data from any page. Strategy:

   - **First contact:** simple expressions (`document.title`) to verify the page loaded.
   - **Explore the DOM** with targeted selectors, one field at a time (`document.querySelector("h1")?.textContent`).
     - **Don't nest** multiple selectors in a single evaluate — can cause 500 errors. Separate into multiple calls.
     - **`document.querySelector(':contains(...)')` causes 500** because `:contains` is not standard CSS. Arrow functions and `:has` work fine. Use standard CSS selectors.

4. **On dynamic pages** (SPA, lazy content, infinite scroll): after creating the tab, use `POST /tabs/{tabId}/wait` on a known selector (e.g. `h1`) or `POST /tabs/{tabId}/scroll` to trigger loading before evaluating.

5. **Readability for articles** (blogs, news, Wikipedia, docs):
   Use `evaluate` + Readability.js (Mozilla, `scripts/camofox/Readability.js`) to extract clean article text, removing nav, sidebar, ads, footer. ~70% token savings vs snapshot. For SEARCH/LIST pages, use snapshot instead.
   On lazy-loading pages: scroll before Readability.

   **Command:**

   ```bash
   # Read Readability.js, build ~94KB JSON payload and inject
   python3 -c "
   import json
   js = open('<skill_dir>/scripts/camofox/Readability.js').read()
   expr = js + '; var a = new Readability(document.cloneNode(true)).parse(); JSON.stringify({title: a?.title, text: a?.textContent, excerpt: a?.excerpt, length: a?.length})'
   json.dump({'userId': '$USER_ID', 'expression': expr}, open('/tmp/rb_article.json', 'w'))
   "

   exec curl -s -X POST "http://localhost:9377/tabs/<tabId>/evaluate" \
     -H 'Content-Type: application/json' \
     -H "Authorization: Bearer $API_KEY" \
     -d @/tmp/rb_article.json
   # Output: { ok: true, result: "...{title, text, excerpt, length}..." }
   # If result is null → fallback to snapshot. Otherwise use text.
   ```

#### Search Macros

Use with `POST /tabs/{tabId}/navigate` passing `macro` and `query`:

| Macro               | Site      |
| ------------------- | --------- |
| `@google_search`    | Google    |
| `@youtube_search`   | YouTube   |
| `@amazon_search`    | Amazon    |
| `@reddit_search`    | Reddit    |
| `@reddit_subreddit` | Subreddit |
| `@wikipedia_search` | Wikipedia |
| `@yelp_search`      | Yelp      |
| `@spotify_search`   | Spotify   |
| `@netflix_search`   | Netflix   |
| `@linkedin_search`  | LinkedIn  |
| `@twitter_search`   | Twitter/X |
| `@instagram_search` | Instagram |
| `@tiktok_search`    | TikTok    |
| `@twitch_search`    | Twitch    |

#### Full workflow example

```bash
# 1. Create tab
exec curl -s -X POST "http://localhost:9377/tabs" \
  -H 'Content-Type: application/json' \
  -d '{"userId":"$USER_ID","sessionKey":"$SESSION_KEY","url":"https://example.com"}'
# → {"tabId":"abc123","url":"https://example.com/"}

# 2. Snapshot (understand structure and refs)
exec curl -s "http://localhost:9377/tabs/abc123/snapshot?userId=$USER_ID"

# 3. If snapshot is sparse → evaluate for raw HTML
exec curl -s -X POST "http://localhost:9377/tabs/abc123/evaluate" \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $API_KEY" \
  -d '{"userId":"$USER_ID","expression":"document.querySelector(\"main\")?.innerHTML || document.body.innerHTML"}'

# 4. Interact: scroll, click, type...
exec curl -s -X POST "http://localhost:9377/tabs/abc123/scroll" \
  -H 'Content-Type: application/json' \
  -d '{"userId":"$USER_ID","direction":"down","amount":500}'

# 5. After each interaction, take new snapshot (refs change!)
exec curl -s "http://localhost:9377/tabs/abc123/snapshot?userId=$USER_ID"

# 6. Click and new snapshot
exec curl -s -X POST "http://localhost:9377/tabs/abc123/click" \
  -H 'Content-Type: application/json' \
  -d '{"userId":"$USER_ID","ref":"e3"}'
exec curl -s "http://localhost:9377/tabs/abc123/snapshot?userId=$USER_ID"

# 7. Structured extract
exec curl -s -X POST "http://localhost:9377/tabs/abc123/extract" \
  -H 'Content-Type: application/json' \
  -d '{"userId":"$USER_ID","schema":{"type":"object","properties":{"title":{"x-ref":"e1"}}}}'

# 8. Close tab
exec curl -s -X DELETE "http://localhost:9377/tabs/abc123?userId=$USER_ID"

# 9. (Optional) Cleanup session (requires API key)
exec curl -s -X DELETE "http://localhost:9377/sessions/$USER_ID" \
  -H "Authorization: Bearer $API_KEY"
```

**Troubleshooting — container down:**

```bash
docker start camofox-browser
# If doesn't exist (⚠️ use --env-file for API keys, bind to 127.0.0.1):
# Create .env file first: echo "CAMOFOX_API_KEY=your-key" > .env
docker run -d --name camofox-browser --restart unless-stopped \
  -p 127.0.0.1:9377:9377 \
  --env-file .env \
  camofox-browser:latest
```

---

### 3. CloakBrowser — Protected sites

For sites with Cloudflare, Akamai, Kasada, DataDome, or when Camofox gets blocked.
Uses `launch()` from the npm package `cloakbrowser`.

Script: `<skill_dir>/scripts/cloak/cloak-fetch.mjs`

```bash
# Simple (markdown output)
exec node <skill_dir>/scripts/cloak/cloak-fetch.mjs "https://example.com"

# Text only (no markdown header)
exec node <skill_dir>/scripts/cloak/cloak-fetch.mjs "https://example.com" --format text

# Raw HTML
exec node <skill_dir>/scripts/cloak/cloak-fetch.mjs "https://example.com" --format html

# With scroll for lazy loading (eBay, Amazon, reviews)
exec node <skill_dir>/scripts/cloak/cloak-fetch.mjs "https://ebay.com/..." --scroll

# Automatic challenge detection (Cloudflare, Akamai, DataDome...)
exec node <skill_dir>/scripts/cloak/cloak-fetch.mjs "https://protected-site.com"

# Proxy + geoip for sites that block datacenter IPs
exec node <skill_dir>/scripts/cloak/cloak-fetch.mjs "https://..." --proxy "socks5://user:pass@proxy:1080" --geoip

# Deterministic fingerprint
exec node <skill_dir>/scripts/cloak/cloak-fetch.mjs "https://..." --seed 12345 --platform windows

# Screenshot (⚠️ writes PNG file — breaks read-only rule)
# Pass URL as positional arg after the script path
exec node <skill_dir>/scripts/cloak/cloak-script.mjs --script "<skill_dir>/scripts/cloak/scripts/screenshot.mjs" "https://example.com"
```

#### cloak-script.mjs — For complex interactions

When click, login, multi-step, or custom data extraction is needed:

```bash
exec node <skill_dir>/scripts/cloak/cloak-script.mjs \
  --script "<skill_dir>/scripts/cloak/scripts/<your-script>.mjs" \
  --proxy "socks5://..." --seed 12345
```

Full guide: `<skill_dir>/scripts/cloak/guida-fetch.md`

---

## Security rules

These rules are enforced by the tooling — do not attempt to bypass them.

- **No internal URLs.** CloakBrowser blocks `localhost`, `127.x`, `10.x`, `192.168.x`, `169.254.x`, and cloud metadata endpoints. DNS resolution is also checked to prevent DNS rebinding attacks.
- **No path traversal.** `--script` must point within the skill directory. Symlinks are resolved before the check. Use `--unsafe` only if you know the risks.
- **Sandbox enabled by default.** User scripts receive proxied Playwright objects with only whitelisted methods. Note: Node.js APIs remain available — for full isolation, a `vm.Context` would be required. Use `--unsafe` to bypass the Playwright sandbox.
- **Rate limiting.** 30 requests/minute by default. Use `--no-rate-limit` to disable.
- **API keys.** Never paste API keys on the command line. Use env vars (`$CAMOFOX_API_KEY`) or `--env-file` for Docker.
- **URL encoding.** Always use `--data-urlencode` with curl — never interpolate raw input into URLs.
- **Docker binding.** Always use `127.0.0.1:` prefix for port mapping. Never expose to `0.0.0.0`.
- **No social media browsing.** Instagram, Facebook, TikTok, LinkedIn, Twitter/X require login — don't attempt with Camofox or CloakBrowser.

### Security-relevant CLI flags

| Flag | Script | Purpose | When to use |
|------|--------|---------|-------------|
| `--unsafe` | `cloak-fetch.mjs`, `cloak-script.mjs` | Bypass SSRF protection, sandbox, and path traversal | Access local/internal services or blocked Playwright APIs |
| `--no-rate-limit` | Both | Disable 30 req/min limit | Bulk scraping (use responsibly) |
| `--verbose` | Both | Include stack traces in errors | Debugging only |
| `--retry <n>` | `cloak-fetch.mjs` | Retry on failure | Unreliable networks |
| `--timeout <ms>` | Both | Navigation/launch timeout | Slow sites (default: 30000) |
| `--wait <ms>` | `cloak-fetch.mjs` | Extra wait after page load | Dynamic content (default: 1000) |
| `--max-chars <n>` | `cloak-fetch.mjs` | Truncate output | Large pages (default: 100000) |
| `--persistent <dir>` | Both | Persistent browser profile | Cookie/session persistence |

---

## Technical reference — Docker containers

Initial setup and diagnostics for SearXNG and Camofox. See `<skill_dir>/docker/setup.md` when needed.