Your site, your rules: new AI traffic options for all customers
Curated from Cloudflare Blog
Managing AI traffic is no longer a binary choice between blocking everything or accepting everything. The recent updates to traffic controls allow practitioners to differentiate between search crawlers, autonomous agents, and training bots. This granularity is critical because agent traffic often behaves differently than traditional bots, consuming resources in unpredictable ways. You can now protect ad-monetized pages specifically, preventing revenue loss from scrapers while still allowing legitimate discovery. For SREs, this means you can tune your infrastructure based on actual bot intent rather than applying broad rules that might inadvertently block valuable search indexing or legitimate user interactions. Start by auditing your current bot management rules to see if they are too blunt. Then, implement specific policies for agent traffic to maintain service stability without sacrificing visibility.
For our second Content Independence Day, we’re giving website owners finer options to manage AI traffic. Instead of a one-size-fits-all block, all customers can now easily distinguish and manage Search, Agent, and Training bots, alongside the new ability to protect ad-monetized pages.
— Cloudflare Blog