Cloudflare has launched a new enforcement tool designed to stop unauthorised scraping of online content by artificial intelligence (AI) firms, a move that could dramatically alter the economics of the internet.
As of July 2025, the internet infrastructure giant now blocks all AI crawlers by default unless they’ve either paid for access or received explicit permission from the content owner.
The company’s new product, Pay Per Crawl, lets publishers charge AI firms a fee every time they crawl their content. If a crawler doesn’t pay, it gets hit with a “402 Payment Required” response, a rarely used HTTP status code that could become the foundation of a new internet revenue model.
Cloudflare powers nearly 20% of the internet. This shift, if widely adopted, could cut off AI companies from training on a vast portion of the web, unless they pay.
Major publishers, including Condé Nast, TIME, BuzzFeed, The Atlantic, Gannett, and others such as Reddit, Pinterest, and Stack Overflow, have already signed on.
Many of them have seen advertising and referral traffic dwindle in recent years as AI-generated summaries and chatbot answers have increasingly replaced the need to click through to the original source.
“This is just the beginning of a new model for the internet,” said Stephanie Cohen, Cloudflare’s Chief Strategy Officer. “The change in traffic patterns has been rapid, and something needed to change.”
That change is reflected in the data. According to Cloudflare, Google’s crawl-to-click ratio has collapsed from 6:1 to 18:1 in the last six months, suggesting users are increasingly finding what they need directly within the search results, especially via AI Overviews. OpenAI’s ratio is far more extreme at 1,500:1.
Historically, search engines indexed the web with the tacit understanding that referrals would follow, generating value for content creators. But AI firms have upended that agreement, lifting vast amounts of content to train their models and offer summarised responses, all while bypassing the original creators entirely.
Some firms go further by ignoring established web standards like robots.txt, which is intended to block unauthorised scraping. Despite publishers’ attempts to draw boundaries, many AI companies insist they haven’t broken any laws.
The legal pushback has already begun. The New York Times sued OpenAI and Microsoft in late 2023 for copyright infringement. Reddit recently took legal action against Anthropic for allegedly harvesting user comments without permission, even though scraping was explicitly prohibited via robots.txt. BBC and Ziff Davis have filed similar lawsuits.
This escalating issue is happening in parallel with massive drops in web engagement. In the U.S., 60% of searches now end without a single click, and click-through rates have plunged by 30% between 2024 and 2025.
Publishers are being squeezed at both ends, their traffic is drying up, and their content is being repurposed without compensation.
Cloudflare’s CEO Matthew Prince framed the initiative as both a defensive and visionary move: “This is about safeguarding the future of a free and vibrant internet.” He pointed at a long-term plan to create a transparent and open marketplace for content access, where AI firms would be required to negotiate fair rates for crawling and training.
For now, Pay Per Crawl is available as a private beta. Cloudflare handles both authentication and payments, serving as the intermediary between web publishers and AI companies.
