back
loading skill details...
Crawl and extract content from websites
Site Crawler Skill Respectfully crawl documentation sites and web content for RAG ingestion. Overview Documentation sites, blogs, and knowledge bases contain valuable structured content. This skill covers: Respectful crawling (robots.txt, rate limiting) Structure-preserving extraction Incremental updates (only fetch changed pages) Sitemap-based discovery Prerequisites # HTTP client pip install httpx
don't have the plugin yet? install it then click "run inline in claude" again.