The biggest names in AI are backing a new way to opt out of scraping

OpenAI, Google, Microsoft, and Anthropic say they will adopt a shared "policy token" signal, proposed by Anthropic's Kevin Kelley within the IETF, that publishers can place in their robots.txt file to apply AI training preferences across multiple AI companies instead of setting them individually. The companies say they will restrict their AI crawlers from accessing a site when they encounter the token, though Microsoft will offer more granular controls for opting out of AI training or AI grounding and notes sites will still appear in Bing search results. Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI says it will implement it in the "coming months."