A Disallow line for GPTBot, CCBot or any other AI crawler in robots.txt does not stop a site's content from being used to train models. Gary Illyes of Google drew that line in remarks to Search Engine Journal in 2026.
robots.txt governs crawling. It tells a bot whether it may fetch pages and pull their contents into a search index. That is the whole of its remit, and it says nothing about training.
So a Disallow for GPTBot means only that the bot will not crawl the pages. If the content has already been collected and folded into a training dataset, the file does not affect that process. It does not recall data from earlier crawls, and it carries no legal force as a ban on use.
The site owner's logic looks sound — close the crawler, close the content — but the two sit on different levels. robots.txt is a directive to bots that choose to honour it, not a licence or a contract. Some AI crawlers ignore it outright, and for them the file is no barrier. Content already downloaded stays downloaded.
For affiliate marketers and SEO teams the file still has a job: keeping search bots out of service sections, limiting crawls of heavy pages and signalling to AI crawlers willing to listen. Content protection is a separate matter, handled through terms of use that spell out a ban on training, authentication and rate limiting on closed sections, and log monitoring to see who is fetching what and how often.
Illyes set out a boundary many site owners had not seen. Crawler access and content use are two different questions, and closing GPTBot in robots.txt answers only the first. Anyone treating that line as protection should check the assumption.
