Google, Microsoft and OpenAI Plan Shared robots.txt Line for AI Training Preferences

Google, Microsoft and OpenAI announced on Friday, October 30, 2026, that they plan to honor one shared robots.txt line for AI training preferences, based on a draft from the IETF’s AIPREF working group. None of the three supports the line today, so site owners cannot use it yet, but the dates now set by Google and Microsoft show when changes begin.

• The three companies announced plans to honor one shared robots.txt line for AI training preferences, based on an IETF AIPREF draft. None supports it yet.
• The token is written as user-agent: __usage_ai-training, and Google will treat an opt-out as a Google-Extended disallow if Google-Extended is not set.
• Microsoft starts honoring its new options February 1, 2027 and drops NoArchive and NoCache as training or grounding signals after March 1, 2027.
• The token does not control crawling. A wildcard group never matches it.
• Google targets 2027 (its documentation says early 2027), while OpenAI expects to follow within months; Anthropic has only said development has started.

Three companies announce a shared AI training line for robots.txt

According to Search Engine Journal, the three companies said on Friday that they plan to honor the same robots.txt line. Google said the same day that it plans to adopt the shared AI training control voluntarily, building on work from the AIPREF working group at the IETF.

The point of the control is efficiency. A site can set its robots.txt AI training preference across several providers in one step, instead of listing each company individually.

Google’s announcement post was written by John Mueller, Google Search Advocate, and is published on Google for Developers. Google says the control reflects feedback from publishers, civil society groups, regulators and other stakeholders.

The mechanism, Google adds, brings the IETF definition of AI training to website owners while multistakeholder discussions continue on the rest of the AIPREF vocabulary. In other words, the training piece ships first, and the broader set of preference terms is still under debate.

How the policy token works in robots.txt

robots.txt is a text file on a website that tells crawlers which parts of the site they may access. AIPREF is the IETF working group on AI Preferences, which drafts machine-readable controls for AI training.

Site owners will use a shared “policy token” in robots.txt to set broad preferences across participating AI providers. The token is written as user-agent: __usage_ai-training, according to Search Engine Roundtable.

No new directive

The shared line does not add a new directive to robots.txt. It is an ordinary rule group, built on the standard robots.txt structure defined in RFC 9309, that uses Allow and Disallow, with the token in the place where a crawler name would normally go, as Search Engine Journal describes it.

The IETF draft gives a sample group that shows how a robots.txt AI training opt-out looks in practice:

Line	What it does
User-agent: __usage_ai-training	Opens a group addressed to the shared policy token instead of a single crawler
Disallow: /	States a training opt-out for the whole site
Allow: /press-kit/	Keeps the press kit folder available for AI training

What counts as AI training

The token’s meaning follows the working group’s vocabulary draft. That draft defines AI Training as using content to modify a generative AI model’s learned parameters. Models that only perform non-generative tasks, such as classification or ranking, are exempt from the definition.

Wildcards, missing groups and old rules

A wildcard group (User-Agent: *) can never match the policy token. If a robots.txt file has no __usage_ai-training group, it indicates no preference at all. Parsers that do not recognize the new group simply ignore it, so existing crawler rules are unaffected.

Rules already in robots.txt files will not be read as part of any shared training preference. A site that wants to take part has to add the group explicitly. Owners who already handle blocking AI crawlers in robots.txt will find that the new group sits alongside those per-bot rules rather than replacing them.

Crawling is a separate matter

The policy token does not control crawling. It states a preference about how content may be used for training, not whether a bot may fetch pages. It also does not change how Googlebot crawls or ranks pages in Google Search.

What site owners should check now

Nothing needs to change today, since none of the three companies reads the shared line yet. A short audit now makes the switch easier later. Start by reviewing your current robots.txt file and listing every group it contains. Confirm that Googlebot is not blocked by a broad rule, because an accidental disallow there removes pages from Google Search, which no AI training setting is meant to do.

Next, note any existing Google-Extended or GPTBot rules. They stay in force after the shared token arrives, and an explicit Google-Extended rule overrides the shared opt-out for Google. If you allow Google-Extended but later add a training opt-out through the policy token, Google follows the Google-Extended rule. Decide which robots.txt AI training preference you want for each provider, then add the shared group once support is confirmed.

What changes for Google-Extended and Gemini

Google-Extended is the AI training control Google has offered since 2023. It is a robots.txt control that governs the use of content for training Gemini models and for grounding in Gemini Apps.

Specifically, it controls whether content can be used to train future Gemini models and whether it can be used for grounding in Gemini Apps and on Vertex AI. Grounding means supplying content from a search index to an AI model when it answers a question.

How the token maps to Google-Extended

Google says that when a site uses the policy token to opt out of training and does not explicitly specify Google-Extended, Google will interpret the opt-out as if the publisher had disallowed Google-Extended. Existing company-specific controls will continue to function alongside the new joint signal.

Sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks when Gemini Apps grounds its answers in their content.

What stays the same for Search

Google-Extended does not affect a site’s inclusion in Google Search. AI Overviews and AI Mode have their own opt-out, the Search generative AI control in Search Console, which rolled out worldwide in August.

The new token therefore does not replace that Search Console control.

Microsoft adds a grounding preference and retires NoArchive and NoCache

Microsoft’s post, by Bing principal product manager Krishna Madhavan, adopts the shared line for training and adds a separate AI Grounding preference, Search Engine Journal reports.

The grounding preference is set only at the page level and governs whether content can help inform AI-generated answers. A meta tag or HTTP header can set the training preference for a single page, overriding robots.txt for that page.

The DisallowExtended option

Microsoft’s new DisallowExtended option allows grounding in Microsoft-created tools such as Copilot and Bing’s AI Mode, but not grounding through Microsoft-provided grounding services used by other businesses, known as Web IQ.

Microsoft says either grounding restriction will limit brand visibility in AI answers and will have no effect on listing in Bing search results.

Old tags are replaced

NoArchive and NoCache tags were used by sites to limit how Copilot uses their content. Microsoft replaces them, for training and grounding, with the two new options. Once Microsoft detects either new preference on a site, it will disregard any NoArchive or NoCache tags still on that site.

Temporary setting in Bing Webmaster Tools

A temporary grounding preference can be recorded in Bing Webmaster Tools. The setting opens December 15 and is available only to sites still using the old tags on that date. Pages without a new preference by March 1 default to allowing grounding until Microsoft recrawls them.

Where OpenAI and Anthropic stand

OpenAI aims to honor the signal in the next few months, and its current robots.txt policy remains in effect. OpenAI uses separate GPTBot and OAI-SearchBot tags, and each setting is independent of the others, according to OpenAI Developers.

A GPTBot disallow indicates crawled content should not be used for training OpenAI’s generative AI foundation models. For search results, OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust.

Neither OpenAI’s blog post nor its crawler documentation, checked October 30, states what happens when a shared-line rule and a GPTBot rule conflict.

Anthropic

As of October 30 there was no Anthropic newsroom posting about the signal. Anthropic’s Kevin Kelley said in a working group email that the company has started development of the feature. Anthropic’s crawler-help page lists three crawl control options: ClaudeBot, Claude-User and Claude-SearchBot.

Kelley’s draft was submitted to the IETF AI Preferences working group on the same date. It is listed as an individual draft, not an official IETF standards-track document. He plans to propose his design as a change to the Content-Usage draft. An earlier Content-Usage rule draft, co-written by Google’s Gary Illyes, was set aside by the working group to make progress on the vocabulary.

Timeline and open questions

Date	Event
December 15	Bing Webmaster Tools temporary grounding setting opens
February 1, 2027	Microsoft begins honoring the new training and grounding options as it crawls. Until then, NoArchive and NoCache remain the signals Microsoft reads
March 1, 2027	Microsoft stops recognizing NoArchive and NoCache as training or grounding options after this date. Pages without a new preference default to allowing grounding until Microsoft recrawls them
May 3, 2027	Kelley’s draft expires unless a new version replaces it

Google’s blog post says it aims to support the control along with other AI providers in 2027 and will provide updates. Its crawler documentation is more specific, saying early 2027. OpenAI expects to move sooner, within months.

Google says the proposal is consistent with the EU Code of Practice for General Purpose AI Model Providers under the EU AI Act, which recognizes robots.txt as a primary method for web publishers to express machine-readable preferences.

Conflicting accounts

Search Engine Roundtable lists Anthropic alongside Google, Microsoft and OpenAI as announcing the control. Search Engine Journal reports no Anthropic newsroom posting as of October 30 and says none of the three companies currently supports the line. This article follows the latter account for Anthropic.

Still unknown is how OpenAI will resolve conflicts between the shared line and GPTBot rules.

Sources

• Search Engine Journal
• Search Engine Roundtable
• Google for Developers
• OpenAI Developers
• Techloy
• trustscan.dev

Photo via Search Engine Journal