Robots.txt Guide: Control Search Engine Crawling
Master robots.txt syntax, directives, and common mistakes.
The robots.txt file tells search engines what to crawl. A misconfigured file can block your entire site.
Syntax
User-agent (which crawler), Disallow (what not to crawl), Allow (exceptions), Sitemap (location).
Common Patterns
Block directory: Disallow: /admin/. Block params: Disallow: /*?sort=.
Critical Mistakes
Never Disallow: / (blocks everything). Don't block CSS/JS. Don't use for hiding pages — use noindex.
Pro Tip
Always test after changes. A typo can block your entire site.
Create a perfect robots.txt with our Robots.txt Generator.
Try Free Tool