Robots.txt Generator
Create a robots.txt with per-bot rules, presets and a sitemap line.
How do you write a robots.txt file that actually works?
Docuboxer's robots.txt generator builds a working file in seconds: pick a one-click preset or start from scratch, add rule groups per user-agent (Googlebot, Bingbot, GPTBot, ClaudeBot...) with their Allow and Disallow paths, then copy or download the result and upload it to your domain root. Everything runs locally in your browser — nothing is uploaded to any server, and there's no signup. Here's the part most generators skip: robots.txt controls crawling, not indexing. If another site links to a URL you blocked, Google can still show that URL in results with no snippet, since it never crawled the page to know what's on it. To actually remove a page from the index, you need a noindex meta tag on the page itself, not a robots.txt rule. It's not a security tool either — malicious bots and scrapers that ignore the standard will read a blocked path just fine, so never rely on it to hide anything sensitive.
How to use Robots.txt Generator
- Pick a starting preset (allow all, block all for a staging site, block AI crawlers, or typical WordPress) or add a rule group from scratch.
- Set the user-agent for each group — use * for every crawler, or specific names like Googlebot, Bingbot, GPTBot or ClaudeBot — and add the paths you want to allow or block.
- Add your sitemap's absolute URL if you have one, and optionally a Crawl-delay per group, keeping in mind Google ignores it entirely.
- Check the live preview, copy the file to your clipboard, or download it as robots.txt and upload it to your domain root (yourdomain.com/robots.txt).
Common use cases
Locking down a staging environment before launch
Apply the "Block all" preset on a staging subdomain so Google never indexes an unfinished version of the site while the team is still building it.
Opting out of AI training crawlers
Use the "Block AI crawlers" preset to stop GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and Bytespider in one click, without touching Googlebot's or Bingbot's access to your pages.
Setting up the standard WordPress rules
Start from the WordPress preset to block /wp-admin/, /wp-includes/ and the plugin and theme folders, while keeping admin-ajax.php reachable — plenty of plugins break without it, even on a mostly-blocked site.
Trimming crawl budget waste on large sites
On catalogs with thousands of URLs — filters, pagination, internal search results — block the low-value paths so crawlers spend their time on the pages you actually want ranked.
Pointing crawlers straight to your sitemap
Add a Sitemap line with your sitemap.xml's absolute URL so Googlebot and Bingbot discover it automatically the moment they read robots.txt, instead of relying only on a manual submission in Search Console.
Frequently asked questions
Does robots.txt stop Google from indexing my pages?
Not by itself. Robots.txt blocks crawling, but Google can still index a blocked URL if another site links to it — it just shows up in results with no snippet, since Google never crawled the content. To actually keep a page out of the index, add a noindex meta tag to the page itself.
Can I block an AI crawler like GPTBot without affecting Google?
Yes. Each rule group only applies to its own user-agent, so you can add a group that sets Disallow: / for GPTBot or ClaudeBot while leaving Googlebot's group untouched. The "Block AI crawlers" preset already does this for the most common ones.
Where does robots.txt need to be uploaded?
Always at the domain root, never in a subfolder: yourdomain.com/robots.txt. A file at yourdomain.com/blog/robots.txt won't be read by any crawler.
Can robots.txt hide private folders or sensitive content?
No — and using it that way backfires, because the file is public. Listing a path in Disallow just tells anyone reading the file exactly where to look. Malicious bots and non-compliant scrapers ignore the file outright. Protect anything sensitive with real server-side authentication instead.
What does Crawl-delay do, and does Google respect it?
Crawl-delay asks a bot to wait N seconds between requests so it doesn't overload your server. Google has ignored it for years — use the crawl rate settings in Search Console instead. Bing and a handful of other engines still partially honor it.
Can I have more than one rule group for the same user-agent?
Yes, though a single group per agent is the usual, clearest setup. The generator lets you add as many as you need and outputs each one as its own block, in the order you created them.