On October 30, Microsoft announced new rules for how site content may be used in its AI products. Site owners will be able to prohibit separately the training of models on their materials and the use of those materials when generating answers in Bing and Copilot. At the same time, pages can still appear in regular search results — that is, refusing the use of content in AI does not mean excluding the site from search.
On the same day, Google said it also intends to support a common mechanism for opting out of AI training. Both companies are orienting toward a standard being developed at the IETF. Its point is that a site owner could set restrictions for several AI providers at once, rather than writing separate rules for each search crawler and each model.
Today such prohibitions have to be implemented point by point: through robots.txt, user agreements and separate technical instructions for different crawlers. A unified mechanism supported at the level of an industry standard should simplify this task and make the rules more transparent for both publishers and developers of AI systems.
The support of the initiative by the two largest players in the search and AI market indicates that the question of control over the use of content for model training is moving from the plane of bilateral agreements into the realm of common technical norms.
