Tavily Search Pro (Native Node)

Intent

Use this skill when you need thorough web research with caching, multi-URL content extraction, credit tracking, or automatic 429 backoff. The skill wraps Tavily's research API in a dependency-free Node.js script and logs queries to disk unless you opt out. Run it when a single quick lookup (use tavily-search-native-node instead) or a known URL fetch (use web_fetch instead) won't cut it. Stop before invoking if the query is privacy-sensitive and sending it to an external API is not approved.

Inputs

Required:

• TAVILY_API_KEY: environment variable holding your Tavily API key. Script reads from process environment only, never from credential files or ~/.openclaw/.env. If missing, the script exits with an auth error.
• task: one of search, extract, stats, or cache. Determines which subcommand to run.
• query_or_urls: search query string (for search task) or space or newline-separated URL list (for extract task).

Conditional:

• privacy_sensitivity: enum (normal, sensitive, client/private, unknown). If sensitive/client/private and you haven't received explicit approval to send this data to Tavily, stop and request approval before proceeding.
• freshness_need: enum (normal, fresh/no-cache, news/freshness-critical). Drives cache behavior. news/freshness-critical forces a live API call.
• depth: enum (basic, advanced). Basic is default and cheaper. Advanced returns more results and costs more credits.
• trusted_domains: optional. Include or exclude specific domains in results.
• max_results: optional integer, default 5. Limit result count. Avoid high numbers unless justified.
• logging_preference: optional. Pass --no-log to skip plaintext query/URL logging. Pass --no-log --no-cache together for maximum privacy on approved sensitive research.
• output_format: optional. Pass --json for structured output instead of human summary.

External connections:

• Tavily HTTPS API endpoints: https://api.tavily.com/search, https://api.tavily.com/extract. Requires outbound HTTPS connectivity. No firewall bypass or proxy setup is performed by the script.
• Local filesystem: ~/.openclaw/cache/tavily-search-pro-native-node/ for cached responses and ~/.openclaw/tavily-search-pro-native-node.jsonl for usage logs. Script will create these directories and files if they don't exist.

Procedure

1. Check environment and arguments.

   • Input: task type, query/URLs, flags.
   • Read TAVILY_API_KEY from process.env. If not set, log error and exit.
   • Parse command-line flags (--no-cache, --no-log, --json, --include, --exclude, --depth, --timeout).
   • Output: validated task, query, flags dict, or error exit.

2. Determine cache eligibility.

   • Input: freshness_need, logging_preference, cache flags.
   • If freshness_need is news/freshness-critical or flag --no-cache is set, skip cache lookup.
   • If freshness_need is normal (default), attempt cache read first.
   • Output: cache_eligible (boolean), cache_path (if applicable).

3. For search task: query Tavily or return cache hit.

   • Input: query string, depth, max_results, include/exclude domains, timeout (default 30s).
   • If cache eligible and cache file exists with valid hash of the request, load and return cached response. Output: cached search results, cache hit signal.
   • Else, construct request JSON with query, api_key, include_domains, exclude_domains, topic (set to "general"), max_results, include_answer, include_raw_content, search_depth. Send POST to https://api.tavily.com/search with timeout.
   • On 429 (rate limit): wait 60 seconds, retry up to 3 times. Log each retry. If all retries fail, return error with retry_exhausted flag.
   • On 401 (auth error): return error and suggest checking TAVILY_API_KEY.
   • On network timeout: return error with timeout flag and suggest increasing --timeout or checking connectivity.
   • On success (200): parse JSON response, write to cache file (cache key = SHA-256 hash of request), append query and result count to usage log (unless --no-log), and output results.

4. For extract task: extract content from URLs.

   • Input: URL list (space or newline-separated), timeout (default 30s).
   • For each URL: construct request JSON with urls array, api_key. Send POST to https://api.tavily.com/extract with timeout.
   • On 429: same backoff and retry logic as search (60s wait, up to 3 retries).
   • On 401: return auth error.
   • On network timeout: return error.
   • On success: append URL count and result to usage log (unless --no-log). Output extracted content for each URL.

5. For stats task: retrieve and display Tavily account usage.

   • Input: none (only API key needed).
   • Make GET request to Tavily stats endpoint (if available) or parse usage from prior API responses (implementation-dependent).
   • If --json flag set, return structured usage object (credits_used, credits_remaining, requests_this_month, etc.).
   • Else, return human-readable summary.
   • Output: usage stats or "stats not available" message.

6. For cache task: inspect or clear local cache.

   • Input: subcommand (info, clear) and flags.
   • If cache info: list all cache files in ~/.openclaw/cache/tavily-search-pro-native-node/, show count, total size, oldest/newest entry. Do not print plaintext request keys (use hashes only).
   • If cache clear: require explicit --confirm flag. Delete all cache files under cache directory. Warn user this is local deletion only and does not affect Tavily's servers. Output: number of files deleted, freed space.
   • Output: cache summary or confirmation of deletion.

7. Format and return output.

   • Input: results from steps 3-6, output_format flag.
   • If --json flag: serialize results to JSON (search: array of results with title, url, content, score; extract: array of url and content pairs; stats: usage object; cache: object with count, size, entries array).
   • Else: human summary (search: numbered list of results with snippet and source URL; extract: content blocks with URLs; stats: readable text; cache: bullet list).
   • Output: formatted string to stdout or JSON to stdout.

Decision Points

Privacy gate (before API call): If privacy_sensitivity is sensitive, client/private, or unknown AND the caller has NOT explicitly approved sending this data to an external API, do not proceed. Return message asking for approval and suggesting --no-log --no-cache if the caller approves. Do not invoke any Tavily API call.

Cache vs. fresh: If freshness_need is news/freshness-critical or --no-cache flag is set, skip cache and make a live API call. If freshness_need is normal (default) and cache_eligible is true and cache file exists with matching request hash, return cached result without API call. Else, make live API call.

Task-specific behavior:

• If task is search and query is empty/null, return error "query required for search task".
• If task is extract and URL list is empty, return error "at least one URL required for extract task".
• If task is stats or cache, ignore query_or_urls input.
• If task is cache clear and --confirm flag is not set, return warning "cache clear requires --confirm flag" and do not delete.

API error recovery:

• 429 (rate limit): backoff 60 seconds, retry up to 3 times. After 3 retries, return error "rate limit exhausted" and advise user to wait or upgrade plan.
• 401 (auth): return error "invalid or missing TAVILY_API_KEY; check environment variable and Tavily account status".
• Network timeout: return error "request timeout; increase --timeout value or check network connectivity".
• Other 4xx/5xx: return API error message and HTTP status.

Logging: If --no-log flag is NOT set, append a line to ~/.openclaw/tavily-search-pro-native-node.jsonl with timestamp, task, query/URL summary, result count, and credits used (if available). If --no-log is set, skip logging. Do not log plaintext queries if logging_preference is sensitive unless query is already in plaintext in the task input (i.e., do not add extra obfuscation; just skip the log line).

Output format: If --json flag is set, return valid JSON to stdout. Else, return human-readable text. Do not mix formats.

Output Contract

Success looks like:

• search task: array of objects, each with title (string), url (string), content (string or null), score (number 0-1). Include cache_hit (boolean) and credits_used (number or null) fields. If cache hit, set credits_used to 0.

• extract task: array of objects, each with url (string), content (string). Include credits_used (number or null).

• stats task: object with credits_remaining (number), credits_used_this_month (number), requests_this_month (number), account_status (string). Or plaintext summary if not --json.

• cache task info: object with cache_count (integer), cache_total_size_bytes (number), oldest_entry_timestamp (string ISO-8601 or null), newest_entry_timestamp (string ISO-8601 or null), cache_dir (string path). Or plaintext list if not --json.

• cache task clear: object with deleted_count (integer), freed_bytes (number), confirm_flag_used (boolean). Or plaintext confirmation message if not --json.

All responses must include a top-level status field: success, error, cache_hit, or not_found.

Data is written to stdout. Cache files are stored as binary or JSON at ~/.openclaw/cache/tavily-search-pro-native-node/<sha256-hash-of-request>.json. Usage log is appended to ~/.openclaw/tavily-search-pro-native-node.jsonl as newline-delimited JSON unless --no-log is set.

Outcome Signal

You know the skill worked when:

1. search task: results array is non-empty, each result has title, url, content, score fields. status field is success or cache_hit. If cache hit, you see "cache_hit": true in the response.

2. extract task: array of URL and content pairs is returned, status is success. Content is readable and matches the target URL.

3. stats task: you see credit remaining count and request count, status is success. Numbers are non-negative integers.

4. cache task info: you see a count of cached files, total size, and timestamp range. status is success.

5. cache task clear: you see a count of deleted files, status is success, and running cache info after shows the cache count decreased.

6. Error cases: status is error, response includes error_code (e.g., "auth_error", "rate_limit", "timeout") and a human-readable message. No partial or malformed data is output.

If logging is enabled, check that ~/.openclaw/tavily-search-pro-native-node.jsonl was appended with a new line (or created if missing) on each task invocation.

If caching is enabled, check that ~/.openclaw/cache/tavily-search-pro-native-node/ contains a new file after a successful search with a SHA-256-derived filename.

Script run with help flag returns command syntax and example invocations without hitting the API.