THE SCOREBOARD FOR WHAT'S NEXTAI + QUANTUM, SCORED. COMPARED. EXPLAINED.
AIQUANTUMSCORE.
← AI & Quantum News

AI / NEWS BRIEF

Major AI providers back a shared robots.txt training preference

OpenAI and Google have committed to a common robots.txt signal for AI-training preferences, while an early IETF draft defines how the cross-provider token should work.

By AIQuantumScore · · Event 2026-10-30

One signal instead of provider-by-provider rules

OpenAI and Google announced plans to support a shared robots.txt policy token for generative-AI training preferences. The draft token is __usage_ai-training: a site can place it in a User-Agent group and use ordinary Allow and Disallow path rules to express its preference. The Next Web reports that Anthropic and Microsoft are also backing the approach. Existing provider-specific controls such as GPTBot and Google-Extended will remain available.

Adoption is promised, not complete

OpenAI says it aims to implement the signal in the coming months. Google targets 2027 and says that, when Google-Extended is not separately specified, a shared opt-out will be interpreted as a Google-Extended disallow. The Next Web reports a February 1, 2027 target for Microsoft. Website operators should therefore keep current provider-specific rules until each company documents production support.

It is still an early standards proposal

OUR ANALYSIS

The October 30 document is version 00 of an Internet-Draft authored by Anthropic's Kevin Kelley. The draft explicitly describes Internet-Drafts as temporary work in progress and expires May 3, 2027. It also says the token expresses a content-use preference rather than controlling crawler traffic; private agreements may override it. The initiative could materially simplify publisher controls if adopted broadly, but it is not an IETF standard or a technical enforcement mechanism today.

More to follow