Robots.txt Generator Online — Free, With AI Presets
Build a correct robots.txt from simple rules — user-agent groups, allow/disallow paths, crawl-delay, sitemap location — or start from a preset for common setups.
- Disallow rules are prefix matches: /api blocks /api and everything under it.
- Sitemap lines must be absolute URLs and are ignored by some older crawlers.
- Robots.txt is a request, not a lock — anything under it is still reachable by URL.
Every conversion in this toolkit runs in your browser with the standard JS APIs. Your input is never uploaded, stored, or logged.
The mental model: groups and prefixes
A robots.txt is a sequence of groups: each User-agent line starts a group, and every Allow/Disallow/Crawl-delay in that group applies to matching crawlers. A crawler uses the group whose name matches it exactly, or the * group when none does. The most specific matching group wins — rules from other groups are not merged in.
Paths are prefix strings, not globs. Disallow: /admin blocks /admin, /admin/login and /administrator alike. The Allow directive resolves conflicts: Google and Bing pick the most specific (longest) matching rule, so Allow: /admin/login inside a group that disallows /admin opens exactly that path.
Presets and honesty about limits
The presets cover the requests that come up weekly: fully open, fully closed (staging), block AI crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) while leaving search engines alone, and a standard site that hides /api and /admin. Edit the generated text freely — it is plain text by design.
Two things robots.txt cannot do: hide content (disallowed URLs can still be linked and indexed without crawling) and enforce anything (it is advisory). For real privacy you need authentication — for de-indexing, meta robots or X-Robots-Tag headers.
Frequently asked questions
Where does robots.txt go?
Exactly one place: the site root — https://example.com/robots.txt. Subdirectory copies are ignored by compliant crawlers.
How do I block AI crawlers but keep Google?
Add separate groups for GPTBot, ClaudeBot, CCBot, and Google-Extended with Disallow: /, then a User-agent: * group that allows the site. The preset labeled 'Block AI crawlers' generates exactly that.
Does blocking in robots.txt remove pages from Google?
No — it prevents crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To de-index, use a noindex meta tag or X-Robots-Tag header on a crawlable page.