Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Cloudflare builds trust ratings for verified crawlers, plans customer thresholds

AI summary

Cloudflare announced a crawler trust evaluation mechanism that scores verified bots over time on whether they follow site crawl rules and comply when asked to slow down, then turns that behavior into a trust rating. The evaluation checks whether crawlers repeatedly access paths restricted in robots.txt and whether they reduce or pause after receiving HTTP 429 responses, focusing on long-term repeated violations rather than isolated errors. The mechanism is currently used internally, with plans to let customers set trust thresholds and to publish ratings on Cloudflare Radar; Cloudflare removed Perplexity's crawler from its verified list in 2025.

In-depth insight

Sites would gain a way to decide how much verified crawler traffic to accept, rather than relying only on identity checks.

Cloudflare公布爬蟲信任評估機制,針對已通過身分驗證的機器人,持續觀察是否遵守網站抓取規則,以及收到降低請求頻率的要求後是否配合,再依長期行為建立信任評等。該機制目前已在內部使用,未來預計開放客戶設定信任門檻,決定如何處理爬蟲流量。

評估對象是Cloudflare的已驗證機器人(Verified Bots),原有機制會確認爬蟲身分及宣稱的用途,並要求不得濫用網站資源。不過,機器人即使通過驗證,仍可能無視網站不希望被抓取的範圍,或在網站要求減少存取後繼續發出大量請求。

評估首先參考robots.txt檔案,網站可在檔案中列出不希望爬蟲抓取的頁面或路徑,Cloudflare會依實際請求,檢查爬蟲是否反覆存取受限制的內容。該檔案用來傳達網站偏好,本身不會阻止爬蟲存取。單次違規可能來自設定錯誤或程式問題,系統因此著重長期、重複發生的違規行為,避免因偶發異常就認定爬蟲不可信。

另一項依據是HTTP 429狀態碼。當網站短時間收到太多請求,可以用HTTP 429要求爬蟲放慢速度,有時也會告知何時適合再次嘗試。Cloudflare會觀察爬蟲收到回應後是否減速或暫停,長期配合能提高信任程度,反覆忽略則會降低評等。即使爬蟲只抓取允許存取的內容,過於密集的請求仍可能造成網站負擔。

Cloudflare在2025年曾將Perplexity爬蟲移出已驗證名單,Cloudflare當時指控該業者無視網站拒絕抓取的要求,並試圖繞過封鎖。

長期觀察結果將整理成信任區間,依爬蟲遵守規則的程度呈現評等,並說明哪些行為影響評估。Cloudflare計畫未來在Cloudflare Radar公開評等,並搭配既有的爬蟲偏好設定與流量管理功能,讓客戶自行選擇信任門檻。

#Cloudflare#Cloudflare Radar#Perplexity#Verified Bots

Original · 2 min readRead the original
Group chat

No comments yet. Start the conversation.