Free Robots.txt Generator
Something not working? Report a problem
About the Robots.txt Generator
Generate a robots.txt file with allow/disallow rules for one or more crawlers, plus a sitemap directive — copy or download it, all in your browser.
How to use the Robots.txt Generator
- 1Set the user-agent (use “*” to target every crawler) and list any paths to disallow or allow.
- 2Add another block if you need different rules for a specific bot, like Googlebot.
- 3Add your sitemap URL so crawlers can find it automatically.
- 4Copy or download the file and upload it as `robots.txt` at your site's root.
Frequently asked questions
Is this robots.txt generator free?
Yes — completely free, with no sign-up and no watermark.
Is my content uploaded anywhere?
No. The file is built entirely in your browser — nothing you type is sent anywhere.
Where does robots.txt need to go?
It must be placed at the root of your domain, e.g. https://example.com/robots.txt — it won't be read from any other location or subfolder.
What's the difference between Disallow and Allow?
Disallow tells a crawler not to access a path (and everything under it). Allow creates an exception to a Disallow rule — for example disallowing “/private” but allowing “/private/public-page”.
Does robots.txt stop pages from appearing in Google?
Not reliably on its own — a disallowed page can sometimes still be indexed (without its content) if other sites link to it. To fully keep a page out of search results, use a “noindex” meta tag instead (see the Meta Tag Generator) and don't disallow it, so crawlers can actually see the noindex tag.