Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Verified Bots

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 1

  1. Cloudflare builds trust ratings for verified crawlers, plans customer thresholds

    Cloudflare announced a crawler trust evaluation mechanism that scores verified bots over time on whether they follow site crawl rules and comply when asked to slow down, then turns that behavior into a trust rating. The evaluation checks whether crawlers repeatedly access paths restricted in robots.txt and whether they reduce or pause after receiving HTTP 429 responses, focusing on long-term repeated violations rather than isolated errors. The mechanism is currently used internally, with plans to let customers set trust thresholds and to publish ratings on Cloudflare Radar; Cloudflare removed Perplexity's crawler from its verified list in 2025.

    iThome 台湾 · 🔥 0

Experience and discussion from the community

Share my Verified Bots experienceAsk about Verified Bots

Nobody has shared their experience with Verified Bots yet.