back
loading skill details...
Extract structured data from websites. Tables, lists, prices, articles, metadata. HTML parsing with caching. Zero external dependencies.
# Smart Scraper Extract structured data from websites with integrated security protections. Supports tables, lists, prices, articles, metadata extraction, plus change monitoring with structured diffs. **Security-first design**: - SSRF protection: blocks private IPs, cloud metadata endpoints, dangerous URL schemes - Cache is **disabled by default** (opt-in via `--cache` flag) - Redirect validation prevents SSRF bypass attacks - Rate-limited requests (100ms minimum interval) - No dynamic code execution (no `eval`, no `execSync`, no arbitrary `require`) - Bounded regex: all HTML pattern matching has content length limits **Cache behavior**: - Disk caching is **opt-in** — use `--cache` flag to enable - Default: every request fetches fresh data - Cache TTL: 5 minutes, max 50 entries, max 10MB total - Cache location: `memory/scraper-cache/cache.json` (in workspace) - Privacy warning displayed when caching is activated - When cache is disabled, no persistent data is written to disk during `--extract` ## ⚠️ Important Warnings ### Watch Mode Persistence `--watch <url>` writes data to disk regardless of cache setting: - Baseline snapshots are saved per-watched-URL - Diff results are stored on each poll interval - These files persist until manually deleted ### HTTP Connections This tool allows both `http://` and `https://` URLs. HTTP connections transmit data in cleartext over the network. - Prefer HTTPS for sensitive scraping targets - HTTP responses may be intercepted or modified in transit - SSRF protection applies to both protocols **Usage modes**: - `--extract <url>` — extract structured data (tables, lists, prices, articles, or all) - `--parse <html>` — parse raw HTML and display structure - `--watch <url>` — monitor for content changes with baseline comparison - `--status` — show cache statistics **Programmatic API**: All functions exported as `module.exports` for testability.
don't have the plugin yet? install it then click "run inline in claude" again.