Cloudflare updated its Disallow AI Training feature to block crawlers collecting data for neural network training while allowing Googlebot through.
The previous implementation was stricter — it blocked all bots, including search crawlers. Sites lost visibility in Google while trying to protect themselves from AI scrapers.
Why Google got an exception
Googlebot indexes pages for search. Blocking it means dropping out of results, which most projects cannot accept.
Cloudflare now divides crawlers into two categories: search bots like Googlebot and Yandex that index content for user queries, and AI crawlers like GPTBot and ClaudeBot that collect data for model training. Disallow AI Training affects only the second group. Search engines continue working as usual.
Bing support delayed
Bing does not yet support Cloudflare's new logic. Microsoft plans compatibility only by early 2027.
If your traffic depends heavily on Bing, enabling Disallow AI Training carries risk — Microsoft's bot may interpret the setting as a full indexing ban. Check your logs after activation.
How it works
Cloudflare analyses the crawler's User-Agent and matches it against a list of known bots. Googlebot is whitelisted automatically. The setting activates through the Cloudflare dashboard under Security. No robots.txt edits required — everything runs at CDN level.
Who needs this
Owners of content projects who want to keep their Google rankings, block use of their text for training ChatGPT, Claude and other models, and avoid editing robots.txt manually for each new AI bot.
For affiliates and arbitrage teams the setting matters less — landing pages have short lifespans and content uniqueness is rarely a priority. But for long-term projects with original material, blocking AI scrapers is becoming part of the strategy.
Should you enable it now
If all your organic traffic comes from Google, yes. No indexing risk.
If you have significant share from Bing, wait until 2027 or monitor crawling manually. The alternative is blocking AI bots via robots.txt, but then you track new User-Agents yourself. Cloudflare simplified the task, but full universal support is not here yet.
