Robots Exclusion Protocol
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- Major AI providers back a shared robots.txt training opt-out
OpenAI, Google, Microsoft, and Anthropic plan to support a shared signal in robots.txt that lets publishers express an AI-training preference across multiple crawlers rather than configure each provider separately. OpenAI and Microsoft aim to add support in the coming months, while Google plans to follow next year; The Verge reports that Anthropic has already adopted the signal. OpenAI says the shared signal will supplement its existing crawler controls, while Meta supports the concept without committing to implementation.
OpenAI · 🔥 32
Experience and discussion from the community
Share my Robots Exclusion Protocol experienceAsk about Robots Exclusion Protocol
Nobody has shared their experience with Robots Exclusion Protocol yet.