A shared AI-training preference is coming to robots.txt. The details still differ

Website owners are getting a proposed shared way to say that their content should not train generative AI models, while leaving ordinary search crawling under separate controls. The practical attraction is a single recognizable preference instead of an ever-growing list of company-specific crawler names.

The proposal is not yet an internet standard, and it is not a technical barrier that prevents copying. Its usefulness depends on providers recognizing the signal and applying their documented rules. Google and Microsoft have described implementation plans, with important differences in timing and scope.

One signal, a narrower purpose

The Internet-Draft published on October 30 defines a policy token called __usage_ai-training. A site can place it in a robots.txt User-Agent group and use the familiar Allow and Disallow path rules. A preference can therefore cover the whole site or particular paths.

This separates the intended use of content from permission for a particular crawler to retrieve it. A publisher might want an article indexed in search but not used to adjust a generative model’s learned parameters. Blocking all retrieval would be a much broader decision, potentially interfering with the very discovery route the publisher wants to retain.

The document is an individual Internet-Draft. Its presence in the IETF’s system does not mean the IETF has endorsed it. The draft also distinguishes its new token from existing crawler rules and allows providers to document how more specific preferences take precedence.

Google keeps its existing control in the picture

In Google’s implementation announcement, support is planned for 2027. The shared preference will interact with Google-Extended, rather than silently replace every existing publisher choice. Where a Google-Extended preference is explicitly supplied, that more specific configuration matters.

That is an important migration detail. Adding a common rule should not be treated as proof that an older, company-specific rule has stopped applying. Publishers need to compare the resulting behavior with their intended policy, especially where they want different decisions for training and content used to support an answer.

Microsoft’s transition has dates worth separating

Microsoft’s announcement introduces distinct training and grounding preferences. The company says these do not change ordinary Bing search eligibility, which remains subject to the usual crawling and indexing requirements.

Microsoft plans to begin applying the new preferences on February 1, 2027, as pages are recrawled. Its older NoArchive and NoCache treatment for these purposes is due to end on March 1. A transition mechanism in Bing Webmaster Tools is scheduled to begin on December 15 for eligible sites already using the older signals.

Grounding controls have another meaningful distinction: the DisallowExtended option permits Microsoft’s own AI experiences while restricting the third-party grounding products described in the announcement. It should not be confused with a blanket refusal of all AI use. Page-level controls and site-level training preferences also operate at different scopes.

For publishers, the dates make this an operational transition rather than a one-line policy statement. A preference has to be deployed, recognized and encountered during recrawling. Microsoft’s temporary measures address part of that gap; they are not a reason to leave the underlying page settings unchanged indefinitely.

The common token could reduce the clerical work of expressing the same decision to several providers. It does not settle compensation, undo past training or compel an uncooperative crawler. Its value will be measured by a more modest outcome: whether a publisher’s choice survives the trip from a configuration file to each provider’s actual use of the content.