back
loading skill details...
When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI…
SEO Technical: robots.txt Guides configuration and auditing of robots.txt for search engine and AI crawler control. When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output. Scope (Technical SEO) Robots.txt: Configure Disallow/Allow, Sitemap, Clean-param; audit for accidental blocks Crawler access: Path-level crawl control; AI crawler allow/block strategy Differentiation: robots.txt = crawl control (who accesses what paths); noindex = index control (what gets indexed). See indexing for page-level exclusions. Initial Assessment Check for project context first: If .claude/project-context.md or .cursor/project-context.md exists, read it for site URL and indexing goals.
don't have the plugin yet? install it then click "run inline in claude" again.