OpenAI, Google, Microsoft and Anthropic back one robots.txt signal for opting out of AI training

What was announced

robots.txt is the small text file at the root of a website that tells automated visitors what they may fetch. Until now, a site that did not want its pages used for AI training had to name each company's crawler or control separately. The new proposal adds one cross-company entry, which the IETF draft calls a "policy token", so a site can state a single training preference that every participating provider reads.

The proposal is an Internet-Draft by Kevin Kelley, dated 30 October and discussed in the IETF's AI Preferences working group. The Verge describes Kelley as a technical lead for standards at Anthropic and reports that Anthropic has already adopted the signal. OpenAI says it aims to implement it "in the coming months". Google says it aims to support it "in 2027", and that when a site opts out through the shared entry without a specific rule for Google-Extended, its existing AI-training control, Google plans to treat that as a Google-Extended opt-out. Microsoft says it will honour the new preferences as it crawls from 1 February 2027.

Meta's communications head Andy Stone voiced support but, according to The Verge, did not say whether Meta will honour the signal. OpenAI calls the arrangement a "voluntary effort", and the companies' existing per-crawler controls stay in place alongside it.

What the signal covers, and what it does not

It covers training only. The draft uses the AI Preferences working group's definition: using content "to modify the learned parameters of a generative AI model". Microsoft says a training choice does not change a site's eligibility for traditional Bing search results.

It is not a crawling block. The draft says the entry "is not a directive for crawling behavior"; a site that wants fewer bot visits still has to use ordinary crawler rules. A general rule addressed to all bots does not count as a training preference either: the draft says wildcard rules "MUST NOT be interpreted" as matching the new entry, and a robots.txt file without the entry expresses no preference at all.

It is also not a licence or a payment scheme. Microsoft says its preferences "do not establish licensing or payment arrangements", and the draft notes that private agreements may override what robots.txt says. The draft itself has no formal standing yet: like any Internet-Draft, it is a work in progress that expires on 3 May 2027 unless it is updated.

Training is not the same as appearing in AI answers

The bigger decision for most businesses sits elsewhere. Microsoft is adding a separate page-level "AI Grounding" preference, covering whether a page can be used to inform answers in Copilot, Bing and Microsoft 365. Its blog warns that limiting or turning off grounding "will significantly reduce your content's visibility in AI-generated experiences". Microsoft adds that its old NoArchive and NoCache tags will no longer be read as training or grounding preferences after 1 March 2027.

Google draws a similar line. It says sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. Google also says the shared signal is consistent with the EU's Code of Practice for general-purpose AI model providers under the EU AI Act, which recognises robots.txt as a primary way for publishers to state machine-readable preferences.

What this means for a small business website

Evoogen's reading: for most small businesses the useful question is not "should AI train on my site?" but "do I want to be cited when someone asks an AI assistant about my service?" The new signal lets those two choices be made separately, at least with the companies that honour it. A plumber or a restaurant that blocks everything AI-related out of caution may also be shutting itself out of AI answers that send customers.

Three checks are worth doing before the 2027 start dates. First, find out what your site's robots.txt already says about AI crawlers; hosting and website platforms sometimes set this for you, and The Verge notes that Cloudflare blocks AI crawlers by default. Second, decide training and visibility separately, and write the decision down so it survives a website redesign. Third, if your site uses Microsoft's NoArchive or NoCache tags, note the March 2027 cut-off. The detailed syntax is in the sources below, and the rules are still being finalised.