llms.txt vs robots.txt
Short answer: robots.txt is a crawl-permission file and llms.txt is a reading list. One doesn't replace the other, and neither one enforces anything.
| robots.txt | llms.txt | |
|---|---|---|
| Status | IETF Standards Track, RFC 9309 (2022) | Community proposal, v2 Aug 2026 (llmstxt.org) |
| Job | Tells crawlers which URLs they may access | Points agents to the content worth reading |
| Location | Must be /robots.txt at the top level of the host (RFC 9309 §2.3) |
/llms.txt or any subpath, e.g. /docs/llms.txt (v2) |
| Format | user-agent groups + allow/disallow rules | Markdown: H1, summary, H2 link lists |
| Enforcement | None. RFC 9309: "These rules are not a form of access authorization." | None. It's a suggestion |
| When it's read | Before automatic crawling | "on demand, when an agent needs information" (llmstxt.org) |
The llms.txt proposal says so itself: "robots.txt and llms.txt have different purposes. robots.txt lets automated tools know what access to a site is considered acceptable… llms.txt information is instead used on demand."
What robots.txt really controls (and doesn't)
- Google: robots.txt "is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page." (Google Search Central)
- RFC 9309 §3: listing paths "exposes them publicly", so don't put secret URLs in robots.txt.
- Matching is case-insensitive on the bot name, and the longest matching path wins (RFC 9309 §2.2.1–2.2.2). If robots.txt returns a 4xx, crawlers "MAY access any resources", and a 5xx means "complete disallow" (§2.3.1.3–4).
AI crawler names, from the operators' own docs
| Operator | Token | What it's for (their words, abridged) | Source |
|---|---|---|---|
| OpenAI | GPTBot | crawls content "that may be used in training our generative AI foundation models" | OpenAI docs |
| OpenAI | OAI-SearchBot | "used to surface websites in search results in ChatGPT's search features" | same |
| OpenAI | ChatGPT-User | user-initiated visits; "robots.txt rules may not apply" | same |
| Anthropic | ClaudeBot | collects content "that could potentially contribute to their training" | Claude Help Center (dated Apr 7 2026) |
| Anthropic | Claude-SearchBot | "to improve search result quality" | same |
| Anthropic | Claude-User | fetches pages when users ask Claude questions | same |
| Perplexity | PerplexityBot | surfaces sites in Perplexity search; "not used to crawl content for AI foundation models" | Perplexity docs |
| Perplexity | Perplexity-User | user requests; "generally ignores robots.txt rules" | same |
| Google-Extended | a robots.txt token only (no separate crawler) controlling Gemini training and grounding use; "does not impact a site's inclusion in Google Search" | Google crawlers list |
Each operator treats its bots independently, so blocking a training bot doesn't block that company's search bot. Google warns: "The HTTP user agent string can be spoofed." (Google's common crawlers, fetched 2026-10-15 07:00 EDT). OpenAI and Perplexity publish IP address lists for their bots, e.g. openai.com/gptbot.json and perplexity.com/perplexitybot.json, as listed in their own docs (OpenAI, Perplexity, fetched 2026-10-15 07:00 EDT).
Illustrative example (we made this up; not advice for your site)
# Allow AI search, opt out of training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Whether you should block training is your call. We don't recommend either way.
Do I need both?
- Keep a robots.txt if you have any crawl preferences at all.
- Add an llms.txt if agents would benefit from a guided path into your content, especially docs.
- Don't list in llms.txt pages that your robots.txt blocks. That sends mixed signals (our opinion).
Check your llms.txt → · Generate one →
Leaflint is independent and isn't affiliated with, sponsored by or endorsed by any company, product or project named on this page. Names are used only to identify them; trademarks belong to their owners.
Guide draft: not indexed until Legal clears it.