The policy token lives in robots.txt and lets publishers set training preferences once for many crawlers.
OpenAI, Google, Microsoft, and Anthropic will adopt a shared policy token that publishers place in robots.txt to opt out of AI scraping. Anthropic standards lead Kevin Kelley proposed the signal through the Internet Engineering Task Force, and the token replaces the current practice of notifying each AI company separately about training preferences.
Robots.txt has long told crawlers whether they may index, archive, or use a site. Publishers now face a steady rise in AI crawlers pulling content into models, and some already push back: the Really Simple Licensing standard lets sites charge visiting crawlers, and Cloudflare blocks AI crawlers by default.
Teams that train or fine-tune models should check robots.txt for the token before assuming a page is fair game. A site that carries it signals a blanket refusal, so pipelines that rely on bulk crawling need a fallback for domains that opt out across every vendor at once.
Support is already uneven: Anthropic enforces the signal today, Google and Microsoft target next year, and OpenAI promises the coming months. Microsoft adds granular controls for training and grounding, and it keeps sites in Bing results regardless of their choice, so publishers should verify each vendor’s behavior before relying on one token.
What matters
- Anthropic already enforces the signal, while Google and Microsoft plan support next year.
- Publishers can now express one preference instead of negotiating crawler rules with each vendor.
- OpenAI says it will ship support in the coming months, so crawler behavior stays uneven for now.
Why it matters
OpenAI says it will ship support in the coming months, so crawler behavior stays uneven for now.
This GenAI News article was prepared in original wording using reporting and materials published by The Verge. Source reference: https://www.theverge.com/tech/1023763/openai-google-microsoft-ai-scraping-standard-opt-out.
Drafted by the GenAI News review pipeline.