robots.txt for AI Crawlers
10 min read · updated August 4, 2026
robots.txt is specified in RFC 9309 and it does exactly one thing: it tells a crawler which paths it is asked not to fetch. It does not control indexing, it does not control training, it cannot remove anything already collected, and it stops nobody who chooses to ignore it. Within those limits it is the correct and standard tool, and most AI-crawler robots.txt files are broken by one rule.
What robots.txt controls, and what it does not
The file lives at the root of an origin — scheme, host and port together, so https://example.com/robots.txt does not govern https://docs.example.com/ and does not govern the same host on a different port. A compliant crawler fetches it, caches it for up to about a day, and applies the group that matches its own name.
| Question | Description |
|---|---|
| Does it stop crawling? | For a crawler that honours it, yes, for the paths you disallow. That is the whole of its stated function. |
| Does it stop indexing? | No. A URL disallowed in robots.txt can still be listed in a search index on the strength of links to it. To keep a page out of an index you must let it be crawled and serve noindex, which is the opposite instruction. |
| Does it stop training? | Only insofar as blocking the fetch prevents collection, and only for that crawler going forward. It has no retroactive effect on a corpus already assembled, and it does nothing about copies of your text on other sites. |
| Does it stop a user-initiated fetch? | Operators differ on whether an agent acting on one person's explicit request is bound by robots.txt at all, and several state that it is not. Read the operator's own documentation for each token. |
| Is it enforcement? | No. RFC 9309 describes voluntary compliance. Enforcement is a firewall, a WAF rule, rate limiting, or authentication. |
The status of the file itself is also specified, and it is worth knowing because a deployment mistake here changes behaviour silently. A 4xx on /robots.txt means no restrictions — crawl everything. A persistent 5xx means the crawler should assume a complete disallow. Serving a 500 during an outage therefore stops crawling; serving a 404 opens the whole site.
The rule that breaks most AI robots.txt files
This is the single most common error in this area and it fails silently in the direction you did not want. A crawler obeys exactly one group: the most specific one that names it. If it has its own group, it does not read the * group at all.
So this file, which is written constantly, does not do what its author believes:
# BROKEN — GPTBot ignores everything below its own group. User-agent: GPTBot Disallow: /private/ User-agent: * Disallow: /private/ Disallow: /internal/ Disallow: /staging/
GPTBot reads its own group, sees one Disallow, and is free to crawl /internal/ and /staging/. The wildcard group is not a base class that specific groups extend; it is a fallback for crawlers that match nothing else. Every group must be complete on its own.
Two more rules from RFC 9309 that decide edge cases. Agent names match case-insensitively and on a prefix of the product token, so User-agent: Google would catch more than you intend. And when both an Allow and a Disallow match a URL, the longest matching path wins, with Allow winning a tie. That is what makes carve-outs work:
User-agent: GPTBot Disallow: / Allow: /blog/ Allow: /docs/ # /blog/post-1 → Allow (6 chars) beats Disallow (1 char) # /pricing → Disallow
The path patterns support * for any sequence of characters and $ for end-of-URL. Crawl-delay is not in RFC 9309; it is a non-standard extension that some crawlers honour and Google states it does not. If you need rate control for a crawler that does not offer a setting, that is a server-side job.
The agent tokens
Below are tokens whose operators publish documentation for them. Each operator’s own crawler page is the authority, and this list moves: new tokens appear, and operators occasionally split one crawler into several with different purposes.
| Token | Description |
|---|---|
| GPTBot | OpenAI. Bulk crawl; content may be used for training. Address ranges are published as a JSON file on openai.com. |
| OAI-SearchBot | OpenAI. Crawls to build the search index that surfaces links in ChatGPT. Blocking this removes you from that surface, which is usually not what someone blocking GPTBot wants. |
| ChatGPT-User | OpenAI. A fetch made because a user asked for that page or that question in a live session. One request, not a crawl. |
| ClaudeBot | Anthropic. Bulk crawl. |
| Claude-User | Anthropic. User-initiated fetch during a conversation. |
| Claude-SearchBot | Anthropic. Crawls to support search results used in answers. |
| PerplexityBot | Perplexity. Crawls to build its index for answer citation. |
| Perplexity-User | Perplexity. Fetch triggered by a specific user request. |
| Google-Extended | Google. NOT a crawler — see the next section. It never appears in your logs. |
| GoogleOther | Google. A general-purpose crawler used for various product and research fetches, distinct from Googlebot. |
| Applebot | Apple. The crawler behind Apple's search features. Address ranges are published. |
| Applebot-Extended | Apple. A use signal in the same shape as Google-Extended, not a separate crawler. |
| Amazonbot | Amazon. Crawls for Amazon products and services. |
| meta-externalagent | Meta. Bulk crawl. Meta also documents a separate token for fetches made on behalf of a user. |
| Bytespider | ByteDance. Widely reported as high-volume; check your own logs for the rate rather than assuming. |
| CCBot | Common Crawl. Not an AI company, but its public corpus is a common input to training sets, so it belongs in this conversation. |
Four complete files
Block bulk training crawls, keep the answer surfaces
The most common actual intent: do not be a training corpus, but do remain citable in assistants that send links back.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: meta-externalagent Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Sitemap: https://example.com/sitemap.xml
Note what is absent: OAI-SearchBot, Claude-SearchBot and PerplexityBot are not blocked, so they fall through to the * group and keep crawling. That is deliberate, and it is the distinction most published block-lists lose.
Block everything AI, accept the consequences
User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: meta-externalagent User-agent: Amazonbot User-agent: Bytespider User-agent: CCBot Disallow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Consecutive User-agent lines with no directives between them form one group applying to all of them, which is specified behaviour and keeps the file short. The consequence of this file is that you disappear from assistant answers that link out, which for many sites is a real cost.
Allow the docs, block the rest
User-agent: GPTBot Disallow: / Allow: /docs/ Allow: /blog/ User-agent: ClaudeBot Disallow: / Allow: /docs/ Allow: /blog/ User-agent: * Allow: / Disallow: /admin/
Everything welcome, junk excluded
This is what this site does, and the reason is not about AI at all. A faceted catalogue generates an unbounded number of query-string URLs, and an aggressive crawler walking them is a load problem before it is anything else. The rule is narrow on purpose: the query strings are disallowed, every real page stays crawlable.
User-agent: * Allow: / Disallow: /models? Disallow: /compare? Disallow: /api/ Disallow: /dashboard Disallow: /chat Sitemap: https://example.com/sitemap.xml Host: https://example.com
/models? matches only URLs carrying a query string, so the catalogue index and every individual model page remain fully crawlable. Blocking the permutations and then never naming the real pages would be half a fix, which is why the Sitemap: line and the file belong together — see sitemaps and getting crawled sooner.
Use signals are not crawl blocks
Google-Extended and Applebot-Extended are the two well-documented examples of a different category, and confusing them with crawlers produces two specific mistakes.
Neither is a crawler. No request ever arrives with Google-Extended in the user-agent header, and you will never find one in a log no matter how long you look. They are tokens that exist only in robots.txt, where they declare a permitted use for content that a different, ordinary crawler already fetched. Disallowing Google-Extended does not reduce Googlebot traffic by one request; it signals that the content should not be used for the generative products the operator names in its documentation.
The two mistakes that follow. First, people block Google-Extended expecting a drop in crawl load and conclude the directive is being ignored when nothing changes — nothing was supposed to change. Second, people block Googlebot intending to opt out of AI use and remove themselves from Google Search entirely, which is exactly the outcome the separate token exists to avoid.
# Out of generative use, still in search. User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: * Allow: /
The resources robots.txt cannot reach
Robots directives in a <meta name="robots"> tag only exist in HTML. For a PDF, a CSV, an image or a JSON endpoint, the equivalent is the X-Robots-Tag response header, which carries the same values and can be scoped to a named agent:
# nginx
location ~* \.(pdf|csv|xlsx)$ {
add_header X-Robots-Tag "noindex, nofollow" always;
}
# Or per agent
add_header X-Robots-Tag "GPTBot: noindex" always;And the standing trap: a page disallowed in robots.txt is never fetched, so its noindex is never read. If you want a URL out of an index, allow the crawl and serve the directive. If you want it not fetched, disallow it and accept that it may still be listed by URL. You cannot have both from one file.
There is also a noai meta tag proposal circulating. It is worth knowing the name so you recognise it, but no major operator documents honouring it, which makes deploying it a statement of intent rather than a control.
Why none of this is enforcement
RFC 9309 describes a voluntary protocol. A crawler that ignores your file is not violating a technical control, because there is no control — it made an HTTP request and your server answered it. Compliance is a choice by the operator, and the operators who publish crawler documentation are self-selecting for the ones who make it.
The uncomfortable corollary is that robots.txt is most reliably obeyed by exactly the crawlers you are least worried about, and publishing a block list tells an operator who does not care precisely which paths you consider valuable. If you need something to actually not be fetched, the tools are:
- Authentication. The only real answer. Content behind a login is content a crawler does not have.
- Address-based blocking at the edge, using published crawler ranges — which works precisely because the honest operators publish them, and therefore has the same coverage problem.
- Rate limiting and challenge rules in a WAF or CDN, which act on behaviour rather than on a claimed identity.
- Terms of service and contracts, which are the only layer that reaches use rather than access. See content licensing between publishers and AI labs.
Verifying the file does what you think
- Fetch it as served, not as written.
curl -sS https://example.com/robots.txt. A framework that generates the file can differ from the file in your repository, and a CDN can serve a cached older copy. - Check the status code and content type. It must be 200 and it must be
text/plain. A framework returning HTML for an unmatched route will hand a crawler an HTML page, which parses to no rules at all. - Read each group as the crawler will. For every named agent, cover the group with your hand and ask whether that group alone expresses your full intent. If it does not, the group is incomplete.
- Cross-check against your sitemap. A URL that appears in the sitemap and is disallowed in robots.txt is a contradiction, and Search Console will report it back to you.
- Then look at the logs. The file states an intent; only the log says what happened. Which is the next page.