Google to support shared robots.txt AI training opt-out in 2027
Google plans to support a shared AIPREF robots.txt token for AI training opt-outs in early 2027, treating it like a Google-Extended disallow. Bing announced separate training and grounding controls with dates from December 15, 2026.
- Google will support the AIPREF training token in robots.txt in early 2027; Bing begins honouring its controls February 1, 2027.
- The IETF proposal is still a draft, and the token covers training only, not AI answers.
- Audit your robots.txt and Google-Extended rules now, and decide your training versus grounding stance.
Ask your own AI: ChatGPT Perplexity Claude AI Mode
On this page
Google said on October 30, 2026 that it will voluntarily adopt a proposed IETF mechanism letting sites opt out of AI training across several providers with one robots.txt rule. Google aims to support it in early 2027. Bing announced its own controls the same day, as reported by Search Engine Roundtable and PPC Land.
What does Google’s new AI training control in robots.txt change?
It lets you set one AI training preference for multiple providers instead of listing each crawler. Per Search Engine Roundtable, the signal is a shared policy token, written as the user-agent __usage_ai-training, based on the IETF’s AIPREF working group. OpenAI and Anthropic are also taking part, according to the sources.
John Mueller wrote that if a site opts out with the token and does not explicitly mention Google-Extended, Google plans to treat it as a Google-Extended disallow. Existing controls keep working, and the Search Console opt-out for Search generative AI remains.
On the Bing side, PPC Land reports separate settings for AI Training, AI Grounding and Limited Grounding. From December 15, 2026 temporary grounding controls appear in Bing Webmaster Tools for sites using NoArchive or NoCache; Bing honours the new preferences from February 1, 2027; the deadline for old-directive sites is March 1, 2027.
Is the AIPREF robots.txt token a final standard yet?
No. It is a draft. Nate Hake, a publisher who has participated in the working group, said on X that it is not finished at the IETF level and significant disputes remain. Wording, scope and syntax could still change, so treat the token name as provisional.
Also note the scope. As PPC Land points out, Google’s token covers training only, while Bing separates training from grounding (retrieving pages to answer a question). Blocking training does not decide whether you appear in AI answers, and publishers still have to read two sets of documentation.
How to prepare your robots.txt for AI training opt-outs
Nothing is live for Google yet, so the job now is planning, not editing. This is our interpretation of the announcements:
- Audit current rules. List which of Google-Extended, GPTBot, ClaudeBot, Applebot-Extended and CCBot you already address.
- Decide training and grounding separately. A common stance is blocking training while allowing citation in answers. Bing now supports that explicitly; PPC Land notes that blocking AI answers can reduce visibility in AI experiences.
- Keep explicit Google-Extended rules. Mueller said they will override the shared token’s default behaviour, so they remain your fine-tuning tool.
- Check Bing dates. If you use NoArchive or NoCache, plan to configure the new settings before March 1, 2027.
- Do not rely on robots.txt as security. Gary Illyes said, per PPC Land, that it is a suggestion, and that Googlebot impersonation is easy. Verify Googlebot by IP and reverse DNS.
- Monitor the IETF draft before hardcoding anything.