Continuous accountability for Verified bots
Curated from Cloudflare Blog
If you operate at scale, you know that a static allowlist is a liability. Cloudflare’s approach shifts the burden from one-time verification to ongoing behavioral compliance. This is critical because a verified bot that ignores rate limits or crawl directives can still degrade your infrastructure or violate data handling policies. By tying access to real-time adherence to robots.txt and HTTP 429 responses, the model aligns bot privileges with actual operational impact. For SREs, this reduces the noise of blanket blocking while preserving control. The concrete takeaway: update your WAF rules to treat bot identity as a dynamic state rather than a permanent credential, ensuring your monitoring captures not just "who" is hitting your endpoints, but whether they are respecting your defined operational boundaries in real time.
Identity alone doesn't guarantee a bot will behave responsibly. We are introducing a system to continuously evaluate whether Verified bots honor site preferences, like robots.txt and HTTP 429s, ensuring they remain accountable over time.
— Cloudflare Blog