Robots.txt does not prohibit AI training — it only asks a crawler not to download pages. Gary Illyes from Google noted that the file determines whether a crawler can fetch content, but not what systems do with the text afterwards. A Disallow line for GPTBot does not remove a site from ChatGPT responses and does not affect pages already downloaded. It applies only to future crawls.
The main issue: AI model training and appearing in AI responses are handled by different bots. OpenAI runs three — GPTBot collects text for training, OAI-SearchBot finds pages for ChatGPT search, ChatGPT-User arrives when a user requests it. Anthropic has a similar setup with ClaudeBot, Claude-SearchBot and Claude-User. Google separated training from search with the Google-Extended token: it does not crawl sites itself but decides whether already downloaded pages go into Gemini training and grounding.
Which bots to block, which to leave open
To block training and stay in responses, block training bots and leave search bots and user-triggered bots open:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Applebot-Extended
Disallow: /
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /
The first group asks not to download for training by OpenAI, Anthropic, Common Crawl and Apple. The second keeps bots open. Google search is unaffected — the site stays in ChatGPT and Claude search. Blocking Google-Extended may cost placement in Gemini responses, so include it only if that matters less than blocking training.
In Russia the options are narrower. Yandex Webmaster advises blocking YandexAdditional or YandexAdditionalBot to keep pages out of Yandex Alice quick answers. The change takes 2–14 days. Yandex has no separate token for training only — blocking YandexBot removes the site from both search and AI responses. Sber does not publish the GigaChat bot name, so it cannot be named in robots.txt.
Which bots ignore robots.txt
Perplexity changes its user agent and ASN when blocked, sometimes without requesting robots.txt. Cloudflare found in August 2025 that Perplexity masqueraded as Chrome on macOS and fetched content from tens of thousands of domains, millions of requests per day. The most compliant was ChatGPT-User — it requested robots.txt and stopped crawling, though OpenAI warns that rules may not apply to it.
Check your own bots through access logs: filter lines with tokens and see which pages they requested after Disallow. A successful response to a blocked section means the bot downloaded the page anyway. If a bot poses as a browser, server or CDN blocking helps — they distinguish bots from browsers by traffic patterns: network and request rhythm.
Other defences
The nosnippet meta tag, which Apple offers, blocks specific content from Siri and Apple search responses to general queries. TDMRep is a W3C protocol that signals a text and data mining reservation. It has legal weight in the EU under Directive 2019/790, but not outside the EU. Since July 2025 Cloudflare has been beta-testing Pay Per Crawl — a site can charge AI crawlers per access.
A general training block has worked at Anthropic since 30 October 2026:
User-agent: __usage_ai-training
Disallow: /
Google plans to support the AIPREF signal in early 2027. IETF is drafting a standard with a Content-Usage rule for robots.txt and an HTTP header, but the value vocabulary is not finalised. Until the standard is adopted, training and responses must be blocked separately by naming each bot.
