firecrawl-crawl

SKILL.md

Bulk extract content from entire websites or site sections with depth and path filtering.

Crawls pages following links up to configurable depth limits and page counts, with path inclusion/exclusion filters to scope extraction

Supports async job polling or synchronous waiting with progress display via --wait and --progress flags

Offers concurrency control, request delays, and JSON output formatting for integration into agent workflows

Part of a four-step escalation pattern: search → scrape → map → crawl, used when single-page extraction is insufficient

firecrawl crawl

Bulk extract content from a website. Crawls pages following links up to a depth/limit.

Prerequisite: crawl requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.

Quick start

# Crawl a docs section
firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json

# Full crawl with depth limit
firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json

# Check status of a running crawl
firecrawl crawl <job-id>