Cloudflare has implemented a schedule to distinguish between standard search crawlers and AI training crawlers. Beginning September 15, 2026, its default settings will block “mixed-use” crawlers—those combining search, agent functionality, or training—from sites with advertisements by default. Existing and new customers will be subject to this policy unless they manually adjust settings.
This decision aims to address publishers’ concerns about intellectual property exploitation. While most seek discoverability through search engines and AI services, they insist on control over how their content is utilized. Cloudflare notes that its default changes will affect all free-tier users, new accounts, and existing users setting up fresh sites.
The initiative impacts AI developers’ access to web data for training. For instance, Google’s crawlers face similar scrutiny. Despite Google’s Extended Robot Exclusion feature allowing opt-outs for training, its core search bot remains active. Cloudflare highlights that Google’s access to content outpaces competitors due to its search-centric approach.
The update transitions Pay Per Crawl to a value-based “Pay Per Use” model, letting publishers charge AI companies when their content generates value. Cloudflare is piloting this with Ceramic.ai and You.com, compensating publishers when their content appears in AI-driven search or premium content access. Other providers can tailor this framework to their needs.
Cloudflare’s strategy also targets efficiency. Data indicates AI crawlers waste 50%+ of resources re-scraping unchanged pages, unnecessarily straining publishers’ bandwidth. By restricting non-purposeful crawling, the platform seeks to streamline operations while compensating content creators fairly.