FREE TOOL
Robots.txt Generator
Generate robots.txt files for your website. Control search engine crawling with allow/disallow rules and sitemap references. Free SEO tool.
Select which search engine crawlers these rules apply to. Use * for all robots.
Paths that search engines should NOT crawl. Leave empty to allow all paths.
Specific paths to allow even if a parent path is disallowed. Allow rules are more specific than disallow.
Full URLs to your XML sitemaps. This helps search engines find all your pages.
Number of seconds crawlers should wait between requests. Not supported by all bots.
How to use this tool
- • Select which search engine bots these rules apply to
- • Add paths you want to block from crawling (one per line)
- • Optionally add specific paths to allow
- • Add your sitemap URLs to help search engines discover pages
- • Set a crawl delay if needed (optional)
- • Copy or download the generated robots.txt file
- • Upload to your website root as robots.txt