Robots.txt Generator

Build a robots.txt file from a simple form. Per-bot allow/disallow rules, crawl-delay, sitemap URL. Validates as you type, copy or download. Always private.

Runs in your browser — nothing uploaded

Rule group 1

User-agentUse * to target every crawler, or pick a specific bot.Disallow pathsOne path per line. Empty list blocks nothing for this bot.Allow paths (optional)Carve out exceptions inside a disallowed parent directory.Crawl-delay (optional, seconds)Honored by Bing, Yandex, and others. Google ignores Crawl-delay — set the rate in Search Console instead.

Sitemaps

Sitemap URLs (one per line)Always use absolute URLs. Listed at the bottom of robots.txt and read by every major crawler.

robots.txt preview

User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Use this free robots.txt generator to assemble a valid robots.txt file from per-bot rule groups, allow/disallow paths, and sitemap URLs. The output updates as you type, with a built-in autocomplete for the bot names you actually care about — Googlebot, Bingbot, GPTBot, ClaudeBot, and more.

Private by design — your data never leaves your device

How to use it

No account, no upload — it all happens on your device.

Start with the User-agent: * group — that's the rule every crawler reads by default.

List the paths to block under Disallow, one per line. Use Allow to carve out exceptions.

Add per-bot rule groups for crawlers you want to treat differently — e.g., a Disallow for GPTBot to opt out of AI training scrapes.

Paste your sitemap URL(s) at the bottom, then Copy or Download the file and upload it to your site root at /robots.txt.

The four directives crawlers understand

Everything in a robots.txt file is one of these.

Directive	Meaning	Example
User-agent	Names the bot the rules apply to. * matches every bot.	User-agent: Googlebot
Disallow	Paths the bot must not crawl. Empty value = block nothing.	Disallow: /admin/
Allow	Exception that re-opens a path inside a disallowed parent.	Allow: /admin/public/
Sitemap	Absolute URL of a sitemap. Listed once, read by every bot.	Sitemap: https://example.com/sitemap.xml

AI crawlers worth knowing about

The newer bots — opt in or opt out deliberately.

GPTBot— OpenAI's training crawler. Block if you do not want your content used to train ChatGPT models.
ChatGPT-User — fired when a ChatGPT user clicks a link to your site (not training). Usually keep allowed.
Google-Extended— Google's separate opt-out for Bard/Gemini training. Does not affect search ranking; works alongside the normal Googlebot rules.
ClaudeBot / anthropic-ai— Anthropic's crawlers (training + retrieval).
PerplexityBot, CCBot (Common Crawl) — wide-net research crawlers; CCBot in particular feeds many downstream datasets.

Common patterns

Goal	What to write
Block all bots from the whole site	User-agent: * Disallow: /
Allow everyone, block nothing	User-agent: * Disallow:
Block one folder for everyone	User-agent: * Disallow: /private/
Block a folder, but allow one sub-path	User-agent: * Disallow: /downloads/ Allow: /downloads/public/
Opt out of AI training only	User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: ClaudeBot Disallow: /

Privacy

The whole file is built in your browser from the form values you enter. Nothing is uploaded, stored, or logged. Copy the result into your editor or Download a robots.txt straight away.

Frequently asked

Where do I put the robots.txt file?

At the root of your domain — exactly /robots.txt. Crawlers do not look anywhere else. Subdomains need their own copy: example.com/robots.txt and blog.example.com/robots.txt are independent files, and one does not apply to the other.

Does robots.txt actually keep pages out of Google?

No — it only blocks crawling. A blocked URL can still appear in Google's index (without a snippet) if it has external links pointing at it. To remove a page from results entirely, allow the crawl and add <meta name="robots" content="noindex"> on the page, or serve the X-Robots-Tag: noindex HTTP header. Use robots.txt for crawl budget control, not for hiding content.

What's the difference between Allow and Disallow?

Disallow blocks the path; Allow carves out an exception inside a disallowed parent. The most specific rule wins. Disallow /admin/ then Allow /admin/public/ lets crawlers fetch /admin/public/page.html but nothing else under /admin/. Empty Disallow ('Disallow:') means 'block nothing for this bot'.