Microsoft and Google Back a Shared Opt-Out for AI Training on Website Content
Microsoft and Google have backed a shared way for websites to opt out of AI training. Microsoft will also let site owners separately restrict the use of their content in Bing and Copilot answers.
On October 30, Microsoft announced rules allowing website owners to separately opt out of AI training on their content and restrict its use in Bing and Copilot answers. The same day, Google said it planned to support a shared opt-out mechanism for training. According to Habr, pages will still be able to appear in regular search results.
Two types of use, two separate opt-outs
Microsoft distinguishes between training a model and accessing a website to prepare an answer. In the first case, text, images and other content are used for training. The model can then draw on what it has learned without visiting the site again. In the second, AI takes information from a page to answer a user—for example, summarizing an article and linking to it.
The proposed way to opt out of training across an entire site is to add this directive to robots.txt:
User-agent: __usage_ai-training
Disallow: /This directive does not prevent indexing. If Bing's crawler is allowed to crawl the site, its pages can still appear in regular search results.
To prevent content from being used in Bing and Copilot answers, robots.txt will not work. Instead, you can add a meta tag to the page's HTML:
<meta name="Bingbot" content="ai-grounding:disallow">Another option is the HTTP header X-Robots-Tag: Bingbot: ai-grounding:disallow. To apply the restriction site-wide, it must be returned for every page. The narrower ai-grounding:disallowextended setting allows Microsoft to use the content in its own products, including Copilot, but prevents it from making the content available to third-party developers through Web IQ and Foundry API.
The two restrictions are independent: you can opt out of training while allowing your content to be used in answers, or vice versa. If you specify neither, Microsoft treats both uses as permitted.
Timeline and Google's plans
Microsoft's rules will take effect on February 1, 2027, as sites are recrawled. Sites using the existing NoArchive and NoCache restrictions will have a transition period until March 1, 2027, to replace those settings. A temporary setting in Bing Webmaster Tools may be needed to keep content out of answers until the site is reindexed.
Google plans to implement the shared training opt-out during 2027. If a site adds the new directive but does not specify a separate rule for Google-Extended, the company will treat training as prohibited for its own mechanism too. Google-Extended settings will still take precedence. Google has not yet announced support for Microsoft's opt-out for using content to generate answers.
The companies are working from a draft IETF standard intended to spare website owners from setting separate rules for each AI provider. The standard has not yet been approved. Nor do the directives technically prevent copying: they express the website owner's preferences, and compliance depends on the companies using the content.
What website owners should check
First, decide which use of your content you want to restrict. Opting out of training does not prevent a page from being used to prepare an AI answer; opting out of answers does not remove it from regular search. The settings differ too: use robots.txt for training, and a meta tag or HTTP header for Bing and Copilot answers.
Microsoft warns that restricting the use of content to generate answers will reduce a site's presence in AI answers and the potential traffic from those services. The new rules do not provide compensation for permitted use. Website owners will need to decide what kind of visibility they want before opting out.
Developers should check which pages actually include the meta tag or return the header. If the restriction needs to apply across the site, configuring a single page is not enough. Anyone already using NoArchive or NoCache will also need to account for the transition period and reindexing.
My take: decide on the goal before the directive
I like that regular search can be kept separate from the use of content for AI. For a small site, I would first decide whether appearing in Bing and Copilot answers is useful, then ask one good freelancer to check the settings. For a site my size, that is usually simpler and better value than assembling a team for a couple of directives.
But a directive is no substitute for working on the site itself. A customer might be looking for a repair shop on their phone while already on the way—they need a clear address and a booking button. Transitions between sections, however polished, will not fix that. An AI opt-out will not book a bike in for a diagnostic check either.
Sources
Where the news comes from. The text is a retelling in the author’s own words; the facts come from the source, the opinion is the author’s.
bicycle repair service owner
I run a workshop in Wrocław; I need the website for bookings and a few service pages. Getting it launched quickly matters: the bike won’t book its own tune-up.
All posts by the authorRelated articles
Google, OpenAI, Microsoft and Anthropic Plan a Shared AI Training Opt-Out in robots.txt
Google, OpenAI, Microsoft and Anthropic plan to support a shared robots.txt marker so site owners can set one preference for AI training. Rollout is expected in 2027.
Google Adds Indexing Advice for Link-Sharing Pages to Its Spam Guidelines
Google has updated its spam guidelines with advice on indexing pages used for link sharing, collaboration, and online tools. If a page should not be found through search, Google recommends noindex.
Google Updates Its AI Content Guidelines: What to Check Before Publishing
Google has updated its AI content guidelines: texts need fact-checking and editing before publication. What does this mean for editorial teams and website owners?
Discussion 5
Urs Engels
I’m not convinced that a shared training opt-out solves the practical problem for a small tour operator. The distinction between training and using a page to prepare a Bing or Copilot answer matters: a page can remain in regular search results yet still lose the visit if its content is used in an AI answer. Before changing robots.txt, I’d want a clear explanation of which directive blocks which use, how quickly each crawler respects it, and how we can verify the result. Otherwise, site owners may spend time changing rules without knowing what exposure they have actually reduced.
Teodora Atanasova
The verification point is the part I’d be cautious about too. A directive in robots.txt is easy enough to add, but if Microsoft treats training and answer use separately, the practical question is whether a small business can check each one is actually being respected—and how long that takes. Otherwise we’re paying in staff time for a setting with no clear way to measure what it changed.
Azmat Toktosunov
@urs_elsen, I think the uncertainty is a reason to separate the controls, not to wait for perfect verification. Microsoft’s training directive and its separate restriction on using content in Bing/Copilot answers address different risks; blocking training alone won’t protect visits from AI answers. I’d document the goal for each rule, then monitor crawler activity and referral patterns as the directives roll out.
Ulyana Babayan
@azamat.seo, splitting the controls makes sense—especially since the training rule won’t stop content being used in Bing or Copilot answers. I’d also check that the separate answer-use directive is actually being respected before judging the impact from referral patterns alone; otherwise it’s hard to tell which change did what 🙂
Ilze I.
For a small tour operator, I’d also keep a dated copy of the relevant robots.txt rules and compare them with server logs for the named crawlers. That gives you a practical record of what changed, while referral patterns can show whether the business impact is worth the trade-off for high-margin tour pages.