Verified Bots
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- Cloudflare builds trust ratings for verified crawlers, plans customer thresholds
Cloudflare announced a crawler trust evaluation mechanism that scores verified bots over time on whether they follow site crawl rules and comply when asked to slow down, then turns that behavior into a trust rating. The evaluation checks whether crawlers repeatedly access paths restricted in robots.txt and whether they reduce or pause after receiving HTTP 429 responses, focusing on long-term repeated violations rather than isolated errors. The mechanism is currently used internally, with plans to let customers set trust thresholds and to publish ratings on Cloudflare Radar; Cloudflare removed Perplexity's crawler from its verified list in 2025.
iThome 台湾 · 🔥 0
Experience and discussion from the community
Share my Verified Bots experienceAsk about Verified Bots
Nobody has shared their experience with Verified Bots yet.