Results are provided for informational purposes only and may contain errors. Please verify important information independently. Use of this tool is at your own risk.
What the robots.txt Generator does
The robots.txt Generator creates the robots.txt file that tells search engine crawlers which parts of a site they may visit. Choose Allow all crawlers or Disallow all crawlers, list any paths to keep out of crawling (one per line, such as /admin/), and add the full URL of your XML sitemap. The rules apply to every crawler (User-agent: *).
Upload the result as a plain-text file named robots.txt at the root of the domain, so it is reachable at https://yourdomain.com/robots.txt. Crawlers read it before anything else. Blocked paths are only listed when the default is Allow, because Disallow all already covers them.
When to use it
- Launching a site and pointing crawlers to its sitemap.
- Keeping crawlers out of admin, cart, search-result or internal API paths.
- Blocking all crawling of a staging or test site.
- Reducing wasted crawl budget on filtered or duplicate URLs.
Worked example
Input
Allow all
Disallow: /admin/, /cart/
Sitemap: https://example.com/sitemap.xmlOutput
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xmlCommon questions about the robots.txt Generator
Does Disallow keep a page out of Google?
Not reliably. It stops crawling, but a blocked URL can still be indexed from links on other sites, without a description. To keep a page out of search results, allow crawling and add a noindex robots meta tag or header instead.
Is robots.txt a security measure?
No. It is a public file and polite crawlers follow it voluntarily. Listing secret paths in it actually advertises them. Protect private areas with authentication.
How do I block AI training crawlers?
Add a separate group for each crawler you want to exclude, for example User-agent: GPTBot followed by Disallow: /. Each company documents the user-agent names its crawlers use.
What if I block my whole staging site but launch with the same file?
Then the live site is blocked from search engines. Check robots.txt as part of every launch; a leftover Disallow: / is one of the most common causes of sudden traffic loss.
Is this tool free?
Yes, it's free to use. No signup, no email required. Get results on this page.
How do I get my result?
Use the tool on this page. Your result appears here; no email delivery is required.