AI Crawlers and robots.txt: A Practical Guide for Publishers

AI companies identify their crawlers by name, and a site's robots.txt file can address each one separately. The most useful distinction is purpose. Some crawlers collect content that may be used to train models (GPTBot, ClaudeBot, CCBot). Others index pages so they can appear in AI search answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). A third group fetches a page when a user asks (ChatGPT-User, Claude-User, Perplexity-User). Google-Extended and Applebot-Extended are different again: they are control tokens, not crawlers.

robots.txt, standardised as RFC 9309, is how a site states its crawling preferences. Compliance is voluntary, though, and the standard says plainly that its rules "are not a form of access authorization".

This guide lists the main AI user agents as documented by the companies themselves. It shows robots.txt syntax with working examples, explains where robots.txt stops, covers other signals such as the llms.txt proposal, and ends with a decision table. It is a practical configuration guide.

Key takeaways

• Several major AI companies run separate agents for training, for search indexing and for user-requested fetches, and each can be allowed or blocked independently.
• Blocking a training crawler does not remove a site from that company's AI search results, and blocking a search crawler can reduce visibility in AI answers.
• Google-Extended and Applebot-Extended control how content collected by Google's and Apple's existing crawlers is used for AI, without affecting search ranking.
• robots.txt is a voluntary standard: user-initiated fetchers may not follow it, and user-agent names can be faked, so server logs and published IP ranges matter.
• A shared, cross-company signal for AI training preferences is emerging in robots.txt, but as of November 2026 it is still a draft being rolled out.

Three kinds of AI crawler, and control tokens

It helps to sort AI agents by what they do with a page.

1. Training crawlers collect public web content that may be used to develop models. OpenAI says GPTBot is "used to crawl content that may be used in training our generative AI foundation models". Anthropic describes ClaudeBot as collecting web content that could contribute to training. Common Crawl's CCBot builds an open repository of web crawl data that is "universally accessible and analyzable by anyone".
2. Search and answer crawlers index pages so an AI product can surface and link them. OpenAI's OAI-SearchBot is used "to surface websites in search results in ChatGPT's search features". Perplexity says PerplexityBot is "designed to surface and link websites in search results" and "is not used to crawl content for AI foundation models".
3. User-initiated fetchers visit a specific page because a person asked a question or requested an action. They are not crawling the web automatically.

Control tokens are a fourth category. Google-Extended and Applebot-Extended never fetch pages themselves. They are names you write in robots.txt to tell Google and Apple how content gathered by their ordinary crawlers may be used.

The main AI user agents

The table summarises each company's own documentation as of November 2026. Companies update these pages, so check the source before relying on a detail.

A few details are worth stressing:

• Google-Extended controls whether content Google crawls may be used "for training future generations of Gemini models" and for grounding in Gemini Apps and Vertex AI. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". It is Googlebot's rules that affect Google Search, "including Discover and all Google Search features".
• Applebot serves two roles: it powers Apple's search features, and its data may also be used to train Apple's models. To keep pages in Apple's search but out of training, allow Applebot and disallow Applebot-Extended. If a robots.txt file has no Applebot rules but does have Googlebot rules, Applebot follows the Googlebot rules.
• OpenAI's settings are independent. A site can allow OAI-SearchBot and block GPTBot. Where both are allowed, OpenAI says it may use one crawl for both purposes.

Because Common Crawl's archive is open to anyone, a CCBot rule affects everyone who later builds on that archive. Our guide to how AI models use data shows how much Common Crawl data features in published training mixes.

robots.txt syntax, with examples

RFC 9309, published in September 2022, is the standards-track specification of the Robots Exclusion Protocol. The essentials:

• The file must be at /robots.txt (all lowercase) in the top-level path of the site, as UTF-8 plain text.
• The file applies to the scheme and host it is served from (the RFC gives its address as scheme:[//authority]/robots.txt), so blog.example.com needs its own file, separate from www.example.com. Anthropic's instructions say to add rules "for every subdomain that you wish to opt out from".
• A group is one or more User-agent lines followed by Allow and Disallow rules. Matching of the agent name (the product token) is case-insensitive, and tokens may contain only letters, underscores and hyphens.
• When rules conflict, "the most specific match found MUST be used", meaning the longest matching path. If an Allow rule and a Disallow rule are equally specific, Allow should win.
• * matches any sequence of characters, $ marks the end of a path, and # starts a comment.
• A crawler that finds no group naming it follows the User-agent: * group. The * group applies only to crawlers that have no group of their own, so rules you want to apply to everyone must be repeated in each named group.

Example: keep content out of training but visible in AI search.

# Training crawlers and AI-use control tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# AI search and answer crawlers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /account/

# Everyone else, including traditional search engines
User-agent: *
Disallow: /account/

Example: protect one section, with an exception.

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /archive/
Allow: /archive/press-releases/

Because the Allow path is longer, /archive/press-releases/2024.html is allowed while the rest of /archive/ is not.

Example: block a file type with the special characters.

User-agent: CCBot
Disallow: /*.pdf$

How errors and caching behave

• If robots.txt returns a 4xx status such as 404, crawlers "MAY access any resources on the server". A missing file means no restrictions.
• If it returns a 5xx server error, crawlers "MUST assume complete disallow". After a long outage, for example 30 days, they may treat the file as unavailable or keep using a cached copy.
• Crawlers should not use a cached copy for more than 24 hours. OpenAI says changes can take about 24 hours to reach its search systems, and Perplexity gives a similar figure.
• Crawl-delay is not part of RFC 9309. Anthropic calls it a "non-standard" extension that it supports; Apple says Applebot does not follow it.

What robots.txt cannot do

robots.txt is an instruction that well-behaved crawlers choose to follow. Several limits follow from that.

• It is not access control. RFC 9309 says so explicitly. Content that must not be fetched needs authentication or server-side blocking.
• User-initiated fetchers may not follow it. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules", because a person requested the page. Anthropic, by contrast, documents Claude-User as controllable through robots.txt.
• Names can be faked. Common Crawl warns that it is "aware of crawlers falsely identifying themselves as CCBot". OpenAI, Perplexity and Anthropic publish IP address lists, and Common Crawl supports reverse DNS checks, so a request can be verified against its claimed identity.
• It looks forward, not back. Anthropic, for example, describes a ClaudeBot block as covering a site's future material. Content collected before a rule was added, or copies of your content hosted on other sites, are outside your robots.txt.
• Blocking by IP can backfire. Anthropic warns that blocking its IP addresses "may not work correctly or persistently guarantee an opt-out", because it stops the crawler reading your robots.txt in the first place.

Other signals beyond robots.txt

A shared AI training preference

A cross-company signal is emerging. Anthropic says that as of 30 October 2026 it has begun recognising a machine-readable preference that lets site owners say "no AI training". The signal follows a draft based on work by the IETF's aipref working group, and Anthropic is rolling it out across its services over time. Google's crawler documentation says Google intends to support the AIPREF shared training preference signal, written as user-agent: __usage_ai-training, in early 2027. In robots.txt it takes the form of an ordinary group:

User-agent: __usage_ai-training
Disallow: /

Treat this as a developing proposal rather than an established standard. It is not part of RFC 9309, and Anthropic says that where a robots.txt file names ClaudeBot, it follows that rule even if the general setting says something different. Company-specific groups therefore remain the most precise control.

Page-level meta tags

Some companies honour page-level directives. Apple, for example, says publishers can use the nosnippet robots meta tag to keep their content out of the AI-generated answers it calls "broad world knowledge answers", while the content remains discoverable through Spotlight, Siri and Safari.

The llms.txt proposal

llms.txt, proposed by Jeremy Howard in September 2024, is a Markdown file, usually placed at /llms.txt at the site root, that gives AI agents a short summary of a site and links to its most useful pages. Its only required element is an H1 title, optionally followed by a blockquote summary and lists of links under H2 headings. It is a community proposal, not a formal standard, and it does a different job from robots.txt. It does not block anything; it guides AI agents to content you want them to use, mainly when they look something up for a user. In its own words, "robots.txt and llms.txt have different purposes."

A decision table for publishers

Practical steps to apply whichever row you choose:

1. Audit your logs. Check which AI agents actually visit, how often, and from which IP ranges.
2. Write named groups. List each agent explicitly rather than relying on User-agent: *.
3. Repeat shared rules. Copy any rules that apply to everyone into each named group.
4. Cover every host. Publish a file on each subdomain that serves content.
5. Check status codes. Make sure robots.txt returns 200, not a 5xx error or an HTML error page.
6. Review the vendor pages each quarter. Agent names and purposes change, and new agents appear.
7. Record the decision. Note what you chose and why, so editorial, commercial and technical teams stay aligned.

Crawler policy is often set alongside content partnerships; our timeline of publisher–AI agreements shows how these arrangements have developed. For how AI search products use retrieved pages, see retrieval-augmented generation explained.

How Druid Learning helps

robots.txt governs what crawlers take from a public website. Many archives were never on the web at all. Druid Learning is an AI archival processing platform, based in Dublin and New York, that turns content archives into AI-ready datasets. Druid processes formats including OCR scans, PDF, XML, EPUB, MP3 and MP4, and its Archive Optimization product turns archives in any format into an indexed, labelled, AI-ready dataset. Through Data Partnerships, Druid prepares an archive, raw or structured, and connects content owners with AI companies.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI documents GPTBot and OAI-SearchBot as separate agents with independent settings. GPTBot relates to training foundation models, while OAI-SearchBot is used to show websites in ChatGPT's search features. A site can disallow GPTBot and still allow OAI-SearchBot. Blocking OAI-SearchBot means a site is not shown in ChatGPT search answers, although OpenAI says it can still appear as a navigational link.

Does blocking Google-Extended affect my Google Search ranking?

Google says it does not. Google-Extended is a control token, not a separate crawler, and it governs whether content Google crawls can be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. According to Google's documentation, it does not affect a site's inclusion in Google Search and is not used as a ranking signal. Google says Googlebot rules are what affect Google Search and its search features.

Do all AI crawlers obey robots.txt?

Not always. robots.txt is a voluntary standard, and RFC 9309 states that its rules are not a form of access authorisation. OpenAI says its rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person requested the page. Some crawlers also impersonate others, so verifying requests against published IP ranges is good practice.

What is llms.txt, and does it replace robots.txt?

llms.txt is a proposal, published in 2024, for a Markdown file, usually at a site's root, that summarises the site and links to its most useful pages for AI agents. It is not a formal standard, and it does not replace robots.txt. robots.txt states which parts of a site crawlers should avoid, while llms.txt points AI agents towards content the site wants them to use.

Sources

• RFC 9309: Robots Exclusion Protocol — RFC Editor
• Overview of OpenAI crawlers — OpenAI
• Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic
• Google's common crawlers — Google Search Central
• About Applebot — Apple Support
• Perplexity crawlers — Perplexity
• CCBot — Common Crawl
• The /llms.txt file — llmstxt.org