Analytics, SEO and growth · Comparison
sitemap.xml vs robots.txt vs IndexNow vs llms.txt
Four ways of talking to crawlers that are easy to confuse. Each does a different job, and none of them forces a page into or out of a search index.
4 options · 7 questions side by side · updated
| Compare | sitemap.xml | robots.txt | IndexNow | llms.txt |
|---|---|---|---|---|
| What it does | Lists the pages that exist | Sets where crawlers may go | Announces changed pages | Summarises a site for AI tools |
| Format | XML | Plain-text rules | An HTTP request plus a key file | Markdown |
| Lives at | /sitemap.xml, or any submitted path | /robots.txt only | Key file on your site; pings go to an API | /llms.txt |
| Who reads it | Google, Bing and most engines | Nearly all reputable crawlers | Bing, Yandex and partners; not Google | Some AI tools and agents; not Google Search |
| Status | Long-established standard | Standard (RFC 9309) | Open protocol from Bing and Yandex | Community proposal |
| Controls indexing | No, it only helps discovery | No, it controls crawling | No, it only notifies | No |
| Effort | Usually generated by the framework | A few lines, rarely changed | A plugin or a small script | Written by hand or generated |
How to choose between sitemap.xml, robots.txt, IndexNow and llms.txt
- Every public site benefits from a sitemap and a robots.txt; both take minutes, and most frameworks generate them.
- Add IndexNow when quick updates in Bing and its partners matter, as for shops, listings or news.
- Add llms.txt if your audience uses AI coding tools or agents, knowing that Google Search ignores it.
The options
- sitemap.xmlA file listing the pages of a site, with when each last changed, so search engines can find them without relying only on links.
- robots.txtA small text file at the root of a site that tells search engine and AI crawlers which parts they may visit and which to leave alone.
- IndexNowA free way to tell search engines the moment a page is added, changed or deleted, instead of waiting for them to crawl again. Bing and several others support it; Google does not.
- llms.txtA proposed Markdown file at /llms.txt that gives AI tools a short, curated guide to a site, with links to its most useful pages. It is a community proposal, not an official standard.
More comparisons
- Google Analytics 4 vs cookieless analyticsBoth answer how many people visit and where they come from. GA4 goes much deeper and ties into Google Ads; cookieless tools trade that depth for privacy, speed and simplicity.
- Meta Pixel vs Conversions APIThe Pixel reports from the visitor's browser; the Conversions API reports from your server. Meta recommends running both and deduplicating, because each catches events the other misses.
- SEO vs answer engine optimisationBoth are about being found. SEO aims for a place in search results; answer engine optimisation aims to be the source an AI assistant quotes. Most of the groundwork is shared.
Crafted in the dark. Shipped to the world.
Tell us what you are building. You get a private project space with a proposal and a line-by-line quote within a day.