AI giants adopt shared opt-out signal for website scraping

OpenAI, Google, Microsoft, and Anthropic will honor a shared robots.txt token, letting publishers set AI training preferences once across crawlers. Creators and site operators gain a simpler control, but Meta has not committed.

What happened

OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable. The shared “policy token” signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It will live within a website’s robots.txt file, which has long signaled which web crawlers can and can’t scrape a site — whether for search indexing, archiving, or AI training.

Currently, sites have to signal their AI training preferences separately to each AI company. With the new control, publishers can apply their AI training preferences across multiple AI companies instead of setting them individually. OpenAI, Google, Microsoft, and Anthropic say they will restrict their AI crawlers from accessing a site if they encounter the shared token. Meta communications head Andy Stone signaled support for the new setting but does not say whether the company will honor it. Microsoft says it will offer granular controls that give publishers the ability to opt out of AI training or AI grounding, the process of pointing an AI model to a verified source of information to reduce hallucinations. Microsoft notes that websites will still appear in Bing’s search results regardless of their preferences. Google previously implemented similar controls. While Anthropic has already adopted the signal, Google and Microsoft plan to support it next year. OpenAI says it will implement it in the coming months.

Key facts

• OpenAI, Google, Microsoft, and Anthropic say they will adopt a shared “policy token” signal proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the IETF.
• The token will live in a website’s robots.txt file, and the companies say they will restrict their AI crawlers from accessing a site when they encounter it.
• Currently, publishers must set AI training preferences separately for each AI company; the new control lets them apply preferences across multiple AI companies at once.
• Anthropic has already adopted the signal; Google and Microsoft plan to support it next year; OpenAI says it will implement it in the coming months.
• Meta communications head Andy Stone signaled support but did not say whether Meta will honor it.

Our analysis

The shift matters because it consolidates a fragmented compliance process. For creators, media operators, and brand sites that publish on their own domains, the current setup likely means maintaining separate crawler permissions across AI companies, with different bot names and opt-out methods. A shared token in robots.txt reduces that maintenance burden and makes a publisher’s preference clearer across several major AI providers at once. The timing aligns with ongoing publisher concern about an influx of AI crawlers scraping sites and feeding content into AI models.

However, the announcement is not a uniform commitment. Meta’s support is limited to a statement from communications head Andy Stone, who says the company looks forward to working with other AI companies on standards and implementation, but does not confirm Meta will honor the token. Microsoft’s granular controls show an important distinction: opting out of AI training is separate from opting out of AI grounding, and neither choice affects Bing search visibility. Google previously implemented similar controls, so operators may need to manage both search appearance and training preferences separately even with a shared token.

The new standard is also not the only response to unauthorized scraping. The source notes Really Simple Licensing (RSL) gives publishers the ability to charge crawlers that visit their sites, while Cloudflare blocks AI crawlers by default. That suggests the shared token is best understood as a cooperative opt-out layer, not a replacement for enforcement, licensing, or infrastructure-level blocking.

What it means for operators

• If you manage a website, blog, newsletter, or community hub, track the rollout windows: Anthropic has already adopted the signal, Google and Microsoft plan support next year, and OpenAI says it will implement it in the coming months.
• Do not assume Meta will honor the token. If Meta crawlers are a concern, keep using existing controls and watch for a clear implementation commitment beyond Andy Stone’s support statement.
• Separate search visibility from AI training. Microsoft and Google offer controls that can opt content out of AI training or AI grounding without removing the site from Bing or Google search results.
• Treat the policy token as one layer, not a full solution. Combine it with other tools such as Really Simple Licensing (RSL) to charge crawlers or Cloudflare’s default blocking, and audit crawler traffic on your owned properties.