Robots.txt Examples: What to Allow and Block

Real robots.txt examples for different website types.

See real robots.txt configurations for e-commerce, blogs, SaaS, and more.

E-Commerce Example

Allow product pages, block faceted navigation URLs, block admin, allow CSS/JS.

Allow all content, block tag archives, block search results, include sitemap.

Allow marketing pages, block app/dashboard, block API docs, include sitemap.

Pro Tip

After creating your robots.txt, test it with our Robots.txt Tester.

Test your robots.txt with our Robots.txt Tester.