Google, Microsoft, OpenAI and Anthropic back one AI-training opt-out

Google, Microsoft, OpenAI and Anthropic said on October 30 that they will support one shared line in robots.txt as a website's preference against having its content used to train generative AI models. The signal is defined in an Internet-Draft by Kevin Kelley of Anthropic, which builds on the definition of AI training agreed in the IETF's AI Preferences (AIPREF) working group.

Until now a publisher that wanted out of AI training had to name each company's crawler or control token separately. None of the four will act on the shared signal at once: Microsoft starts on February 1, 2027, Google in early 2027, OpenAI "in the coming months", and Anthropic says it has started building support.

What a site owner writes

The draft adds no new directive. A site writes an ordinary robots.txt group whose User-Agent line names a "policy token", __usage_ai-training, instead of a crawler. To refuse AI training for the whole site:

User-Agent: __usage_ai-training
Disallow: /

Paths work as they do for any crawler. The draft's own example refuses training everywhere except a press kit:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Four rules in the draft decide what that means in practice:

• A User-Agent: * group never counts. A file with no __usage_ai-training line expresses no preference at all, "as is true of every one published today", Kelley wrote in his note to the working group.

• The token is a preference about use, not a crawl rule. The draft says it "is not a directive for crawling behavior"; blocking a crawler still takes a group for that crawler's own name.

• "AI training" means using content "to modify the learned parameters of a generative AI model", the working group's definition. Training models used only for non-generative tasks, such as classification, ranking or scoring, is outside it.

• Specific expressions should take precedence over general ones, and private agreements may override the signal.

Who starts when

Each company set its own date, and none is a hard date except Microsoft's.

Sources: Kevin Kelley's IETF post; OpenAI; Microsoft Bing; Google's crawler documentation, all October 30, 2026.

Anthropic's commitment came in Kelley's note, not a company blog post. "It'll take us some time to add support across our services," he wrote. Google's announcement said it aims to support the control "in 2027"; its crawler page narrows that to early 2027.

What the signal does not cover

Search. The token does not touch search indexing. Microsoft said the preferences "do not change eligibility for traditional Bing search results", Google says Google-Extended does not affect inclusion in Google Search, and OpenAI keeps search under its separate OAI-SearchBot crawler.

Grounding, at Microsoft. Using a page to inform an AI answer at the moment it is written is a separate choice. Microsoft added its own page-level AI Grounding preference, set by meta tag or HTTP header, with no robots.txt version. Its "DisallowExtended" setting keeps a page in Microsoft's own products, such as Copilot, while withholding it from grounding Microsoft sells to others.

The company announced the change on X:

Grounding, at Google. Google will read the shared opt-out "as if the publisher had disallowed Google-Extended", unless the site names Google-Extended itself. Google's documentation says Google-Extended governs both training of Gemini models and grounding in Gemini Apps and Vertex AI. A site that wants to stay in Gemini's answers while refusing training elsewhere has to say so to Google separately: "Websites that specifically allow Google-Extended are eligible to appear in responses," Google wrote.

Existing tokens. GPTBot, ClaudeBot and Google-Extended keep working. OpenAI said its existing crawler controls "will continue to apply and be respected", and Kelley said Anthropic's support "complements our existing crawler controls rather than replacing them". Microsoft is the exception: the new preferences replace its NoArchive and NoCache tags for training and grounding, and after March 1, 2027 it will no longer read those tags as either.

What it means for publishers and software companies

For a site that wants out of AI training entirely, the work shrinks to two lines that four of the largest model makers have said they will read. The catch is timing. By SaasBuzz's count, Microsoft's start is 94 days away, and its deadline for sites that rely on NoArchive or NoCache is 122 days away. Google and OpenAI have given no day at all.

The signal's standing is also provisional. The draft is an individual submission that, its IETF Datatracker page says, is not endorsed by the IETF, and it expires on May 3, 2027, 185 days after publication, unless revised. Kelley said he is offering it as input to the working group's attachment work. The draft thanks 46 contributors by name, including four from Google, four from Microsoft and two from OpenAI, and the Search Central post says Google will keep working to standardize it through the working group.

Software companies with documentation, help centers or blogs face the same choice as news sites, with one trade-off made explicit by Microsoft: refusing grounding "will significantly reduce your content's visibility in AI-generated experiences", it warned, while refusing training does not change search eligibility. The shared token settles only the training half.

What to do now

• Add a __usage_ai-training group if you want to refuse training across participating providers; it changes nothing for crawlers that do not read it.

• Keep existing GPTBot, ClaudeBot and Google-Extended rules, and check that they do not contradict the new group.

• If Gemini answers matter to you, add an explicit Google-Extended group: Google will otherwise read the shared opt-out as a Google-Extended disallow, which its documentation says also covers grounding in Gemini Apps.

• If you use NoArchive or NoCache for Bing, publish the new training and grounding preferences before March 1, 2027, and record a temporary grounding preference in Bing Webmaster Tools, which starts accepting them on December 15.