Robots.txt Generator

Robots.txt is a text file placed in the root directory of a website that tells search engine crawlers which pages or sections of the site should not be accessed. It's part of the Robots Exclusion Protocol.

• Don't block important pages that should be indexed
• Use specific paths rather than broad disallows
• Always include your sitemap URL
• Test your robots.txt with Google's testing tool
• Use crawl-delay to prevent server overload from aggressive bots

Where should robots.txt be placed?

At the root of your website, e.g., https://example.com/robots.txt

Does robots.txt prevent indexing?

No. It only prevents crawling. Pages can still be indexed if linked from other sites. Use noindex meta tag for preventing indexing.