On 10 30, OpenAI, Google, and Microsoft each announced support for a cross-vendor AI training preference signal: websites can add a group in robots.txt with the policy token __usage_ai-training as the User-Agent, using Disallow / Allow directives by path to express "do not allow / allow for AI training." Participating vendors will all honor this signal, eliminating the need to write separate rules for each vendor's crawler. The proposal originates from a draft submitted to the IETF by Kevin Kelley, Anthropic's Head of Standards Technology, and is based on the AIPREF working group's definition of "AI training."
Confidence: High (95) · Event time 2026-10-30
Rationale
- Previously, websites had to block GPTBot, Google-Extended, ClaudeBot, and others individually; missing one effectively meant granting permission. Google states that if a website uses this token to declare no training allowed but does not separately specify Google-Extended, Google will treat Google-Extended as prohibited. Existing vendor-specific controls remain valid, allowing sites to make a unified statement and then individually permit specific vendors.
- Implementation timelines vary: According to The Verge, Anthropic has already adopted it; Google plans to support it alongside other vendors in 2027; Microsoft states it will enforce it during crawling starting 2 1, 2027; OpenAI says "in the coming months." Meta expressed support for the direction but did not state whether it would comply.
- Microsoft is simultaneously adjusting its own rules: adding page-level "AI training" and separate "AI grounding" (grounding, i.e., citing web pages in answers) preferences. Grounding can be set to Disallow or DisallowExtended (restricting use to Microsoft's own products only). These settings do not affect a page's eligibility for ranking in Bing search results. The old NoArchive / NoCache directives will no longer be read as training and grounding preferences after 3 1, 2027.
Usage
- Websites that do not want to be used for training can add the following to robots.txt: User-Agent: __usage_ai-training followed by Disallow: / on the next line (individual directories can be reopened using Allow). The wildcard User-Agent: * will not be interpreted as this token. The full draft is available on the IETF datatracker (draft-kelley-aipref-training-preference-signal).
- Site owners using Bing / Copilot should also review the migration instructions on the Bing Webmaster Blog: sites using NoArchive / NoCache must switch to the new settings before 3 1, 2027. Starting 15 12, temporary grounding preferences can be registered in Bing Webmaster Tools.
Risks
- This is merely an individual submission to the IETF (version -00); the draft itself states it has not received IETF endorsement. The rest of the AIPREF working group's vocabulary is still under discussion.
- It is only a preference declaration, not access control: non-participating vendors and crawlers can ignore it; it also does not involve licensing fees or payments.
- The original OpenAI request was rejected (403); OpenAI's specific commitments and timeline are derived from RSS summaries and reporting by The Verge.
Open · Google Search Central Blog · Bing Blog · IETF Draft draft-kelley-aipref-training-preference-signal · The Verge