Robots.txt vs sitemap.xml: what each file actually does

Robots.txt restricts crawling. Sitemap.xml suggests URLs worth crawling. Neither file controls indexing. That single distinction settles most arguments about these two files, including the one you are probably here for. A Disallow rule is not a delete button, and a sitemap entry is not an indexing request — what actually keeps a page out of Google is a noindex directive, and what merges duplicates is a canonical tag. You can build and sanity-check both files with the robots.txt generator and the sitemap.xml generator, which run entirely in your browser and never upload your URL list anywhere.

Side by side

• Robots.txt — plain text, must live at the host root (https://example.com/robots.txt), read before crawling, tells bots where they may not go. Public to anyone.
• Sitemap.xml — XML, can live anywhere on the host, read during crawling, tells bots which URLs exist and when they changed. Also public, and also just a hint.

They overlap in exactly one place: the Sitemap:line inside robots.txt, which points crawlers at your sitemap with an absolute URL. That line is the reason people assume the files are two halves of one system. They are not — they answer different questions, and neither answers "should this be in the index?"

What robots.txt actually controls

Scope first, because this trips up more sites than syntax does: a robots.txt file governs one scheme, host and port. The file at https://example.com/robots.txt says nothing about https://shop.example.com/ or the http:// version. Every subdomain needs its own.

The grammar is small. User-agent opens a block, Disallow and Allow define path rules, and Google supports two wildcards: * for any sequence of characters and $ to anchor the end of a URL. When rules conflict, the most specific match wins, which is how Disallow: /cart/ plus Allow: /cart/help does what you expect.

Two limits worth internalising. First, robots.txt is not access control — it is a request that well-behaved crawlers honour and hostile scrapers ignore. Second, it is a public file, so a line reading Disallow: /internal-admin/ advertises the path you were trying to hide. If content must stay private, it needs authentication, not a text file.

What sitemap.xml actually does

A sitemap is a discovery aid. It earns its keep on large catalogues, brand new domains with few inbound links, and pages that internal navigation buries. On a tidy 30-page site where everything is two clicks from the homepage, it changes almost nothing — which is fine, since it costs almost nothing to publish.

The hard limits are worth knowing: 50,000 URLs and 50 MB uncompressed per sitemap file. Beyond either threshold you split the file and publish a sitemap index that lists the parts. Every URL should be absolute, on the same host, return 200, and be the canonical version. Listing redirects, 404s, noindex pages or non-canonical duplicates sends mixed signals and clutters your Search Console coverage reports for no benefit.

As for the optional fields: lastmod is the only one Google says it uses, and only when it reflects genuine content changes. Rewriting it nightly across every URL teaches Google to discount it. changefreq and priority have been ignored for years. Setting your homepage to priority: 1.0 is not a ranking lever; it is decoration.

The mistake that costs the most: crawling is not indexing

Crawling means fetching the page. Indexing means storing it so it can be served in results. Conflate the two and you end up with the classic failure mode: a URL blocked in robots.txt that still shows up in Google, ranking on its bare URL with no title, no snippet, and a line saying no information is available for this page. It is still in the results and now it looks broken.

The trap has a second, sharper edge. If a page carries a noindex tag and you block it in robots.txt, Google never sees the noindex, because reading it would require crawling the page you just blocked. The correct sequence is: allow crawling, add noindex (meta robots in HTML, or an X-Robots-Tag header for PDFs and other non-HTML files), wait for the URL to drop out of the index, and only then add a Disallow if you want to save crawl budget.

Pick the tool by the outcome you want:

• Stop bots fetching it (crawl budget) → robots.txt Disallow.
• Keep it out of search results → noindex, on a crawlable URL.
• Consolidate near-duplicates → rel=canonical.
• Keep strangers out entirely → authentication. None of the above will do it.

Three situations where this gets decided

Staging environments. A blanket Disallow: / on staging.example.com is better than nothing, but it will not save you when someone links the staging URL from a public ticket. HTTP authentication across the whole environment is the real fix. The mirror-image disaster is more common than the original: shipping staging's Disallow: / to production. Check the live file on launch day, every time.

Faceted navigation on e-commerce. Colour, size, price and sort parameters combine into effectively unlimited URLs that burn crawl budget and produce nothing worth ranking. Blocking those parameter patterns in robots.txt is the standard answer, with one caveat: if the faceted URLs are already indexed, blocking them freezes them in the index instead of clearing them. Do noindex first with crawling allowed, then block once they are gone.

AI crawlers. GPTBot, ClaudeBot, CCBot and Google-Extended each identify themselves with a documented user-agent and honour robots.txt rules aimed at them. Whether to block them is a publisher decision with real trade-offs on both sides — blocking protects your content from being used for training and retrieval, and also removes you from the answers those assistants give. Google-Extended is the narrow one: it governs use of your content in Google's AI products, not your ranking in ordinary Search. The robots.txt generator ships a preset with the current AI user-agents so you are not copying stale lists off a forum. The usual disclaimer applies — a crawler that ignores the file will ignore it.

A short pre-flight checklist

• robots.txt returns 200 at the domain root, and contains no accidental Disallow: /.
• No CSS or JS needed for rendering is blocked — Google needs those files to understand the page.
• No URL is simultaneously noindex and blocked in robots.txt.
• The sitemap lists only canonical, 200-status URLs, and is referenced both by a Sitemap: line and in Search Console.
• No changefreq or priority; lastmod only where it is truthful.

Done properly, both files are boring: written once, re-checked after big deploys, otherwise ignored. Nearly every problem attributed to them comes from asking them to do a job that belongs to noindex or a canonical tag.

Frequently asked questions

Related tools

• Robots.txt generator — Per-user-agent rules, AI crawler presets and the sitemap line.
• Sitemap.xml generator — Valid XML that respects the 50,000 URL limit.
• Meta tag generator — Where noindex and the rest of the page-level signals actually belong.
• URL slug generator — Clean URLs before they ever reach your sitemap.

You might also like: free SEO tools that do not ask for your data and what a URL slug is and how to write a good one.