Anthropic runs three crawlers with separate jobs, and only one of them collects text for model training. Sites that block all three shut themselves out of Claude's web search results.
Crawl Readiness checked the top 1,000 sites in October 2026. Of the 569 that have a robots.txt at all, 63 block Claude-SearchBot — one in nine. That is the crawler the site's visibility in Claude answers depends on.
Who does what
The training bot gathers text for AI model training. The search bot indexes pages for Claude's web search. The user bot arrives when someone asks the model to read a specific page. Each can be blocked separately in robots.txt, so training can be shut off without cutting the site off from search: block the training bot and the other two keep working.
A page reaches Claude's sources only if the search Claude uses can see it and the site admits both the search and the user bot. Block the search bot and the page is not in the index. Block the user bot and it cannot be fetched when a query comes in. The training bot barely matters here.
What 63 of 569 shows
One site in nine with a robots.txt has closed Claude-SearchBot. For an affiliate or arbitrage site that is a direct loss: the user asks a question, the model goes to its index, and the material is not there. The traffic never arrives, and no report shows it missing.
The fix is to open robots.txt, find the Anthropic entries and check which bot is blocked. If it is the search or user bot and Claude answers are wanted, lift the block. If the aim was only to keep text out of training, blocking the training bot alone would have done it. The list of Anthropic bots changes, so the file is worth rechecking whenever content or SEO strategy does.
