Google, OpenAI, Microsoft and Anthropic Plan a Shared AI Training Opt-Out in robots.txt
Google, OpenAI, Microsoft and Anthropic plan to support a shared robots.txt marker so site owners can set one preference for AI training. Rollout is expected in 2027.
Google, OpenAI, Microsoft and Anthropic plan to support a shared mechanism in robots.txt that will let site owners state their preferences for using content to train AI. As Habr News reported on November 2, 2026, owners will be able to set a common rule instead of configuring each company separately. Rollout is planned for 2027: for now, this is a plan, not a tool already in use across all four companies.
How the shared rule will work
The proposal builds on the work of the AIPREF group at the Internet Engineering Task Force (IETF). It calls for a policy token in robots.txt that would let site owners express one rule about using their material for training across the companies taking part. This does not mean giving up provider-specific settings: individual developers’ existing tools will continue to work.
For Google, one such tool is Google-Extended. If a site owner opts out of training through the shared token without setting Google-Extended separately, Google will treat the shared opt-out the same way as an opt-out through Google-Extended. Other AI developers may take a similar approach. But “may” does not mean we already know how every participant will handle every combination of shared and provider-specific rules.
The idea is that a site owner can either set a restriction for all participants or specify a different choice for particular providers. That reduces the number of separate instructions, but owners will still need to understand exactly what they are restricting and which individual settings are already in place. We have already covered Microsoft and Google’s support for a shared AI training opt-out on the blog; the key question now is how the shared rule will work alongside existing tools.
Training, AI answers and search are not the same thing
The proposed token concerns the use of content for training. It should not be read as a blanket ban on AI products accessing a site in any way. An earlier Microsoft announcement mentioned by the source shows why these uses need to be distinguished: site owners will be able to separately block the use of their material to train models and its use to generate Bing and Copilot answers. Their pages can still remain in regular search results.
There is another nuance for Google. Sites that explicitly allow content use through Google-Extended may appear in responses from Gemini apps and receive traffic through grounding. That is a possibility, not a promise of a particular number of visits. The announcement does not tell us how many visitors any given site would gain or lose by choosing one rule over another.
Google says the initiative responds to calls from publishers, civil society organisations and regulators for a common approach. The report also says the proposal aligns with the Code of Practice for General-Purpose AI Models under the EU AI Act, which recognises robots.txt as the main machine-readable way for site owners to express their preferences. The mechanism still needs further development and standardisation at the IETF, so an announcement of support should not be mistaken for final implementation rules.
What site owners should do before rollout
I would start by checking the settings already in place for individual companies: the shared mechanism will not automatically replace them. Then the business needs to decide what restriction it wants—one rule for all participants, or exceptions for particular providers. That decision is best made with an understanding of how much regular search results and potential visibility in AI answers matter to the site, rather than simply because a single setting sounds appealing.
It makes sense to hold off on technical changes until the final rules are available. The report does not provide the token’s final syntax or explain in detail how every combination of shared and individual directives will be interpreted. For now, a developer would be better off documenting the current settings and agreeing on the desired restrictions with the site owner. Once the mechanism is clarified, they will have a baseline for comparing the new configuration, without treating one shared token as the answer to every question.
My view: define the boundaries before making promises
As a shop owner, I can see the value of one rule instead of several, but I am not ready to treat easier configuration as an SEO benefit. I look at my advertising and organic traffic reports myself, so I care not only about where the opt-out is set, but also about what happens to visits. AI answers may change click behaviour, but customers still need to find and compare products. The source offers no calculation that would let me estimate the token’s effect on my shop, and I do not want to replace one with guesswork.
I would want the boundaries clearly defined: what the shared token restricts, how a separate directive interacts with it, and which ways for material to appear in answers remain separate choices. “Set it once” sounds convenient, but without those details it is hard to compare with the tools we already have. When rollout begins, I will need clear rules and a way to compare them with my site’s results—not just a shorter robots.txt.
Sources
Where the news comes from. The text is a retelling in the author’s own words; the facts come from the source, the opinion is the author’s.
sporting goods store owner
I run a small store and review the paid and organic traffic reports myself. Before changing CMS, I want to see not just a migration plan but also an estimate of what we’ll gain in return.
All posts by the authorRelated articles
Microsoft and Google Back a Shared Opt-Out for AI Training on Website Content
Microsoft and Google have backed a shared way for websites to opt out of AI training. Microsoft will also let site owners separately restrict the use of their content in Bing and Copilot answers.
Google Adds Indexing Advice for Link-Sharing Pages to Its Spam Guidelines
Google has updated its spam guidelines with advice on indexing pages used for link sharing, collaboration, and online tools. If a page should not be found through search, Google recommends noindex.
Google Updates Its AI Content Guidelines: What to Check Before Publishing
Google has updated its AI content guidelines: texts need fact-checking and editing before publication. What does this mean for editorial teams and website owners?
Discussion 5
Elena Kraft
Since rollout is only planned for 2027, what should a small store do in the meantime—keep using provider-specific settings like Google-Extended, or wait for the shared token? I’d want to avoid accidentally blocking normal search visibility while opting out of AI training.
Lauri K. Post author
I wouldn’t wait for the shared token. Keep any provider-specific settings you already need, but check each rule’s scope before changing it: Google-Extended is separate from Google Search crawling, so opting out of training does not by itself mean blocking ordinary search visibility. The shared mechanism is still a 2027 plan, and the details may change; for a small store, the practical move is to document the current settings and have whoever maintains your site verify them after changes. I’d also avoid adding rules for every provider unless you’re clear on what each one blocks.
Inés García
@lauri.market, I’d add a simple pre-change check: test the robots.txt rules against a few important product and category URLs, then record whether each provider can still crawl them. That makes it easier to catch an accidental block before it affects search or other site functions.
Ilze Ozola
Just to clarify: Google-Extended is a separate control for Google’s AI uses, so changing it doesn’t mean you have to block regular Google Search crawling. I’d keep provider-specific settings in place for now, but check each bot’s documentation before changing rules—especially for a small shop where product pages need to stay easy to find. Are you mainly concerned about training use, or also about your products appearing in AI answers?
Lauri K. Post author
That distinction is exactly why I’d treat training use and product visibility as separate decisions. For our shop, I’d keep ordinary search crawling open and only change a provider-specific training control after checking its scope; the shared token is still a 2027 plan, so waiting would leave the current settings unchanged, not make them safer. I’m less certain about AI answers: do you see a practical way to opt out of training without also reducing the chance that products are surfaced in those answers?