hasdata

fetch real-time web data via the hasdata CLI. use one subcommand per API. flags and enums are derived from the live schema at api.hasdata.com/apis. reach for hasdata when the user asks about current information (prices, news, reviews, listings, job postings, social profiles, contact details) or wants to scrape a URL, summarize a page, verify a link is live, monitor a price over time, or extract structured data from arbitrary web pages.

intent

use hasdata to ground answers in current web data instead of relying on stale training data. invoke this skill when the user asks "what does google say about X", "is this still true", "summarize this URL", "find me businesses that do X", "what's the going rate for Y", "track this product's price", "pull the latest news on Z", or explicitly requests web scraping. hasdata wraps Google, Bing, Amazon, Shopify, Zillow, Redfin, Airbnb, Yelp, YellowPages, Indeed, Glassdoor, Instagram, Google Maps/Trends/News/Images/Flights/Events APIs, and can scrape arbitrary URLs with JavaScript rendering, proxying, and AI-powered field extraction.

inputs

required:

• HASDATA_API_KEY environment variable. user must run hasdata configure once, paste their key, and it saves to ~/.hasdata/config.yaml (mode 0600). every future call picks it up automatically. if missing, the CLI errors with no API key configured.
• hasdata binary. if not installed, run curl -sSL https://raw.githubusercontent.com/HasData/hasdata-cli/main/install.sh | sh.

optional:

• --api-key KEY flag to override the env var (rarely needed).
• --timeout DURATION to override the default 2-minute per-request timeout.
• --retries N to change max retries on 429/5xx (default 2).
• --proxy-country Y or --gl Y flags for geolocation (useful for geo-targeted SEO checks, geo-blocked content, checking how a page renders in country Y).

subcommand selection context:

• user intent (search, news, shopping, maps, jobs, real-estate, social, URL scraping).
• whether the user wants broad results (e.g., google-serp) or deep details on a single item (e.g., amazon-product, zillow-property).
• whether the user needs to fan out from a list of items to per-item details (e.g., search results to individual property pages).

procedure

1. install and configure (one-time setup)

input: none (or user's hasdata API key if not yet configured).

output: CLI installed, config file at ~/.hasdata/config.yaml with mode 0600.

steps:

• check if hasdata is installed: command -v hasdata.
• if missing, run the install script: curl -sSL https://raw.githubusercontent.com/HasData/hasdata-cli/main/install.sh | sh.
• if this is the user's first time, have them run hasdata configure, paste their API key when prompted, and confirm it's saved.
• on all future calls, the key is picked up automatically.

2. pick the right subcommand based on user intent

input: user intent (e.g., "web search", "latest news", "product prices", "maps", "real-estate", "jobs", "scrape URL", "summarize article").

output: one subcommand name (e.g., google-serp, google-news, web-scraping).

refer to the subcommand matrix below. if the user's intent is ambiguous or multi-faceted, start with the broadest match (e.g., google-serp for "what does google say about X", google-news for "latest news on X"). if the user later asks for deeper details on a single item (e.g., a specific property, product, or job listing), escalate to the matching -property, -product, or -job subcommand.

user intent	subcommand
web search ("what does google say about...")	google-serp (full) or google-serp-light (cheap, single page)
latest news	google-news
AI mode SERP	google-ai-mode
shopping / product prices (broad)	google-shopping
shopping / product prices (Amazon-specific)	amazon-search or amazon-product
shopping / product prices (Shopify)	shopify-products
immersive product page	google-immersive-product
maps / places / reviews / photos	google-maps, google-maps-place, google-maps-reviews, google-maps-photos
Yelp / YellowPages local data	yelp-search, yelp-place, yellowpages-search, yellowpages-place
real-estate listings (broad)	zillow-listing, redfin-listing, airbnb-listing
real-estate single property deep dive	zillow-property, redfin-property, airbnb-property
jobs (broad)	indeed-listing, glassdoor-listing
jobs (single job deep dive)	indeed-job, glassdoor-job
Bing search	bing-serp
trends	google-trends
images	google-images
flights	google-flights
short videos	google-short-videos
events	google-events
Instagram profile	instagram-profile
Amazon seller	amazon-seller, amazon-seller-products
scrape a specific URL	web-scraping (supports JS rendering, proxies, markdown, AI extraction, screenshots)

3. construct the command with flags

input: subcommand name, required flags (e.g., --q for search, --asin for product), optional flags (e.g., --num, --gl, --proxy-country).

output: a valid hasdata CLI command.

rules:

• flag names are kebab-case; the CLI maps them to camelCase before sending to the API.
• booleans defaulting to true have a negation: --no-block-ads, --no-screenshot, --no-js-rendering. do not set both --block-ads and --no-block-ads; it errors.
• any flag ending in -json accepts inline JSON (--extract-rules-json '{"title":"h1"}'), a file (--extract-rules-json @rules.json), or stdin (cat rules.json | hasdata web-scraping ... --extract-rules-json -).
• repeatable kv flags split on the first =, so values with = survive: --headers User-Agent=foo --headers Cookie=session=abc.
• list flags accept either repeats or comma-joined: --lr lang_en --lr lang_fr or --lr lang_en,lang_fr.

4. run the command with output handling

input: constructed hasdata command, desired output format (JSON, markdown, screenshot, file).

output: raw response bytes or file written to disk.

steps:

• always pass --raw when piping to jq (skips pretty-print and TTY detection).
• use --pretty only for human-readable terminal output.
• use -o, --output FILE to write binary (e.g., screenshots) or large responses to a file instead of stdout.
• add --verbose to log the outgoing URL and X-RateLimit-* headers to stderr if debugging rate limits or auth.
• example: hasdata google-serp --q "espresso machine" --num 10 --raw | jq -c '.organic_results[] | {title, link, snippet}'.

5. parse the response with jq

input: JSON response from hasdata, jq filter expression (e.g., .organic_results[], .price, .reviews[].rating).

output: extracted fields, normalized to user's request.

steps:

• if unfamiliar with the API's response shape, run a single call with --pretty to inspect the structure.
• write the jq filter based on the schema (e.g., .organic_results[] for SERP organic results, .jobs[] for job listings, .price for product price).
• pipe the filtered output into downstream processing (CSVs, summaries, enrichment, alerts).

6. handle pagination and multi-result aggregation

input: large result sets, need for multiple pages or deep dives on individual items.

output: all results aggregated into a single structure (file, stdout, or database).

steps:

• for SERP / news / listings with many results, use the --num flag (e.g., --num 100) to fetch more in a single call. check the specific subcommand's help (hasdata <api> --help) for pagination options.
• for deep dives on individual items (e.g., pulling full details on each product from a search), use xargs to fan out: hasdata google-shopping --q "laptop" --num 10 --raw | jq -r '.products[].id' | xargs -I {} hasdata amazon-product --asin {} --raw.
• for monitoring a price over time, loop the command on a schedule (cron, systemd timer) and persist the .price and .timestamp to a file or database.

7. interpret exit codes and troubleshoot

input: exit code from hasdata, stderr output.

output: error diagnosis and next action.

interpretation:

• exit 0: success. proceed to output parsing.
• exit 1: user / CLI-input error (missing required flag, bad enum value, missing API key). tell the user to re-read the error message and correct the command.
• exit 2: network error (timeout, unreachable, DNS failure). retry with --retries N or --timeout DURATION.
• exit 3: API returned 4xx (auth, quota, validation). check that the API key is correct and not expired; verify the request format against the help text (hasdata <api> --help).
• exit 4: API returned 5xx. hasdata already retried; the API is down or rate-limited. wait a few seconds and retry.

decision points

if the user has not yet configured hasdata:

• error: no API key configured.
• action: direct the user to run hasdata configure, paste their key, and confirm it's saved to ~/.hasdata/config.yaml.
• never invent or guess an API key.

if the user asks "what does google say about X" or "is this still true" or "what's the latest on Y":

• invoke google-serp or google-news first. do not rely on training data cutoffs or hallucinate stale answers.
• if the user needs deeper details on a single item from the results (e.g., a specific business, property, or article), escalate to the matching -property, -place, or -job subcommand.

if the user asks to "summarize this article" or "TL;DR this URL":

• use web-scraping --url <URL> --output-format markdown to extract clean markdown (strips ads, nav, scripts).
• pipe the markdown into a summary prompt instead of asking the user to copy-paste.

if the user asks to "verify this link" or "is this site real":

• use web-scraping --url <URL> --no-block-resources to fetch the page and check the status code.
• add --screenshot to visually inspect the page (output a PNG file with -o screenshot.png).
• or use google-serp --q 'site:example.com' to check if the domain is indexed and has any presence.

if the user asks to "find alternatives to X" or "what are competitors for Y":

• use google-serp --q "X alternatives" or google-shopping --q "X competitors".
• fan out the results to deep dives if the user wants more details on each competitor.

if the user asks "what's the going rate for X" or "how much does Y cost":

• use google-shopping --q "<product>" to get a broad price distribution across retailers.
• use amazon-search --q "<product>" to focus on Amazon.
• extract prices with jq '.products[] | .price' and calculate min/max/avg if needed.

if the user asks for a "phone number" or "address" for a business:

• use google-maps-place --place-id <ID> or yelp-place --id <ID> (already have the place ID from a prior search).
• do not guess from training data. always verify with a live query.

if the user asks "are people happy with X service" or "is X reputable":

• use google-maps-reviews --place-id <ID> --sort lowest to sample negative reviews first.
• use glassdoor-job --id <ID> to pull employer reviews if asking about a company's reputation as an employer.

if the user asks "what's the salary range for Y role in Z location":

• use indeed-listing --q "<role>" --l "<location>" or glassdoor-listing --q "<role>" --l "<location>".
• extract salary fields with jq '.jobs[] | .salary' and calculate the distribution.

if the user asks to "find homes/apartments matching X criteria":

• use zillow-listing --q "<query>" or redfin-listing --q "<query>" or airbnb-listing --q "<query>" with location and filter flags (e.g., --price-min, --price-max, --bed-count).
• refer to references/real-estate.md for the full filter catalog.

if the user asks "recent sold comps near X" (real-estate):

• use zillow-listing --type sold --keyword "<location>" --days-on-zillow 12m.
• filter to a specific price range with --price-min and --price-max if needed.

if the user asks to "track this product's price" over time:

• loop amazon-product --asin <ASIN> or google-shopping --url <URL> on a schedule (cron job, Lambda, systemd timer).
• extract .price and .timestamp and write to a file or database.
• set up an alert if price drops below a threshold or exceeds an upper bound.

if the user asks "what's trending around X":

• use google-trends --q "<topic>" for relative search interest over time.
• use google-news --q "<topic>" for recent headlines and news velocity.

if the user asks to "find businesses near me that do X":

• use google-maps --q "X" --ll "@LAT,LNG,12z" (substitute actual lat/lng or a place name).
• extract .local_results[] with place IDs.
• fan out to google-maps-place --place-id <ID> for contact details, hours, reviews on each result.

if the user asks "how does this look in country Y" (geo-targeting, SEO, geo-blocked content):

• use --gl Y on SERP commands (e.g., google-serp --q "..." --gl fr for France-localized results).
• use --proxy-country Y on web-scraping to render the page from country Y's IP.

if the user asks to "pull structured data from this page" without writing CSS selectors:

• use web-scraping --url <URL> --ai-extract-rules-json '{"price": {"type": "number"}, "title": {"type": "string"}, ...}'.
• the API uses AI to extract fields based on semantic rules, not brittle CSS selectors.

if the user has a "list of items → per-item details" workflow:

• pattern: initial search command (e.g., google-shopping) produces IDs/URLs.
• pipe IDs through xargs into the matching -property, -product, or -place deep-dive subcommand.
• example: hasdata google-shopping --q "laptop" --num 10 --raw | jq -r '.products[].id' | xargs -I {} hasdata google-immersive-product --product-id {}.

if the user asks to "find this person's role / employer / LinkedIn / followers":

• start with google-serp --q '"<Person Name>" linkedin' (quoted name improves precision).
• the organic result title typically contains Name , Role at Company | LinkedIn and the snippet carries location, headline, follower/connection count.
• often the SERP answers the entire question without opening the LinkedIn profile page. only escalate to instagram-profile or dedicated profile scraping if SERP misses the field.

if the user asks "what is company X doing? where's their HQ? who works there?":

• use google-serp --q "<COMPANY>" to surface the .knowledge_graph block, which pre-extracts founder, HQ, founded year, parent company, employee count range.
• use google-news --q "<COMPANY>" for recent activity and press coverage.
• for specific facts, use targeted SERP: --q '"<COMPANY>" headquarters', --q '"<COMPANY>" funding', --q 'site:linkedin.com/company "<COMPANY>"'.

if the user asks to "find emails for company X" or "personal email for person Y":

• start with `google-serp --q '"@