Cloudflare’s new default settings will block AI crawlers from ad-supported sites, forcing payment for publisher content

Cloudflare has announced a significant policy change that will reshape how AI companies access web content. Starting September 15, 2026, the company’s default settings will block so-called “mixed-use” crawlers — bots that combine traditional search indexing with AI training and agentic services — from crawling any pages that host advertisements. The move, announced Wednesday, gives the AI industry a clear deadline to separate their crawlers by purpose.

The change applies automatically to new Cloudflare customers, new sites set up by existing customers, and all existing free-tier customers. Site owners can adjust these settings manually if they wish to allow mixed-use crawlers. For most publishers, however, the default will now protect their content from being used for AI training without compensation.

Why Cloudflare is drawing a line

Cloudflare co-founder and CEO Matthew Prince framed the decision as a necessary step to preserve a sustainable internet ecosystem. “Now that the majority of traffic on the Internet is non-human, we must go further and act faster,” Prince said in the announcement, referencing the recent milestone where bot traffic surpassed human traffic for the first time — a shift not expected until 2025.

Prince noted that most website owners want their content to be discoverable through search and AI services, but they also want protections against having their intellectual property given away for free. The new default settings are designed to encourage AI companies to clearly separate search indexing from training and agentic use, giving publishers more control.

Google in the crosshairs

Cloudflare specifically called out the “world’s largest search engine” — a clear reference to Google — as having access to “about 2x more information” than other AI companies. The company argues that Google makes it difficult for customers to remain discoverable without also being used for AI training. Google has previously pushed back against this characterization, pointing to its Google Extended bot, which allows site owners to opt out of having their content used for training and AI products like Gemini Apps and Vertex API. Google says opting out via Google Extended does not affect a site’s inclusion in Google Search. However, the company’s primary Googlebot crawls for Search, including AI features such as AI Overviews and AI Mode, which Cloudflare views as a form of mixed-use crawling.

Pay Per Crawl evolves into Pay Per Use

Cloudflare has been developing tools to give publishers more leverage in the AI era. Earlier this year, the company launched a marketplace called Pay Per Crawl, which lets websites charge AI bots for scraping. That product is now evolving into “Pay Per Use,” which will allow publishers to charge AI companies when their content creates value — not just when it is fetched. The change could also reduce wasted bandwidth: Cloudflare’s data suggests that over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages.

To put the model into practice, Cloudflare is initially working with two partners: Ceramic.ai and You.com. When a publisher opts in, they are paid when their content appears in Ceramic’s AI search results or when You.com accesses a piece of their premium content. Cloudflare says other AI companies can customize this model to fit their own workflows.

What this means for publishers and AI companies

For publishers, the policy shift offers a clearer path to monetizing their content in an era where AI models increasingly rely on web-scraped data. For AI companies, it introduces a new cost and operational complexity: they must either separate their crawlers by function or negotiate payment terms with publishers. The September 2026 deadline gives the industry roughly two years to adapt, but the pressure is already building.

The broader implication is a potential restructuring of how the web’s data economy works. If Cloudflare’s default becomes an industry standard, AI companies may find themselves paying for access to content that was previously free for the taking — a shift that could have major financial and strategic consequences.

Conclusion

Cloudflare’s new policy represents a clear attempt to rebalance power between publishers and AI companies. By blocking mixed-use crawlers from ad-supported pages by default, the company is forcing the AI industry to either separate its bots by function or pay for the content it uses. The September 2026 deadline provides a runway for adaptation, but the message is unmistakable: the era of free, unrestricted web scraping for AI training is coming to an end. Publishers now have a powerful ally in their fight for fair compensation.

FAQs

Q1: When does Cloudflare’s new policy take effect?
September 15, 2026. The change applies automatically to new Cloudflare customers, new sites set up by existing customers, and all existing free customers. Site owners can adjust settings manually if they wish.

Q2: What are “mixed-use” crawlers?
Mixed-use crawlers are bots that combine traditional search indexing with AI training and agentic services. Cloudflare’s new policy blocks these crawlers from ad-supported pages by default, forcing AI companies to separate their crawlers by purpose.

Q3: How does Pay Per Use work?
Pay Per Use is an evolution of Cloudflare’s Pay Per Crawl marketplace. It allows publishers to charge AI companies when their content creates value — for example, when it appears in AI search results or is accessed as premium content. Cloudflare is initially working with Ceramic.ai and You.com to implement the model.