Cloudflare’s AI crawler policy marks a turning point for the open web — shifting from unrestricted AI scraping to permission-based access and usage-based compensation. Source: Image generated by author using AI
For thirty years, the deal between the web and the crawlers that read it was simple. You let Googlebot in, Google sent you traffic, and that traffic paid your bills. Nobody wrote this down anywhere. It didn’t need to be written down. It worked.
That deal is dead now, and Cloudflare just wrote the obituary.
On July 1, 2026, Cloudflare announced that starting September 15, AI training bots and AI agent crawlers will be blocked by default on ad-supported pages across its network, unless a site owner explicitly turns them back on. Search crawlers stay allowed. Everything else starts at zero. Cloudflare sits in front of roughly a fifth of the web, so this isn’t a niche policy tweak. It’s a default that becomes the internet’s default the moment enough sites inherit it.
If you’re building anything that touches retrieval, agents, or model training, this changes what “the web” means as a data source. Not eventually. In about ten weeks.
What actually changed
Cloudflare used to offer one blunt instrument: a “Block AI Bots” toggle. On or off. Every AI crawler got treated the same, whether it was indexing your page for search, answering a live user query, or hoovering up text to train the next foundation model.
That toggle is gone as the primary control. In its place, Cloudflare now splits bot traffic into three categories:
- Search — traditional indexing, the Googlebot-style crawl that feeds a search results page.
- Agent — bots fetching content live, on behalf of a user, to answer a specific question right now. Think a Claude or ChatGPT session pulling a page mid-conversation.
- Training — bulk collection aimed at building or fine-tuning a model, with no immediate user-facing task attached.
A site owner can now allow one, block another, and leave the third untouched. That granularity is available to every Cloudflare customer, including free tier.
Then comes the part that actually forces the industry’s hand. On September 15, new domains onboarding to Cloudflare, new sites from existing customers, and all free-tier accounts that haven’t opted out will default to: Search allowed, Training and Agent blocked, on any page that carries ads. Existing paid customers keep their current settings unless they change them, but the center of gravity for the entire web is about to shift toward a default no.
The clause that actually forces a decision
Here’s the mechanism that matters most for anyone paying attention to how this plays out: mixed-use crawlers.
Most crawlers today don’t cleanly separate “I’m indexing this for search” from “I’m also feeding a training pipeline.” Googlebot, Applebot, and Bingbot all blend functions in a single crawl. Under the new rules, if a bot can’t or won’t declare which category a given request falls under, it gets treated as guilty of all of them. Block Training on your site, and any crawler that mixes Training with Search gets blocked outright, even for the Search part.
That’s a deliberate squeeze. Cloudflare wants crawler operators to run separate, declared bots for separate purposes. A search-only crawler that stays search-only sails through. A crawler that quietly reuses the same fetch for model training gets caught in the net it built for itself.
Cloudflare has been fairly direct about who this is aimed at. Without naming Google outright, the company’s announcement noted that the largest search engine has access to roughly twice as much web content as other AI companies, because staying visible in Google Search effectively means also feeding Google’s AI products. Google’s counter is that Google-Extended already lets publishers opt out of AI training without losing search visibility. Cloudflare’s counter to that counter: Googlebot itself, the one that actually matters for search ranking, still crawls for AI Overviews and AI Mode, and there’s no way to allow one without the other short of blocking Googlebot entirely. That’s the asymmetry Cloudflare is trying to legislate away.
Why this is happening now
The trigger isn’t philosophical. It’s a traffic problem.
The old value exchange, in one shape:
Publisher writes content
↓
Search engine indexes it
↓
User clicks a link
↓
Publisher gets traffic → ad revenue / subscriptions
The AI-answer shape:
Publisher writes content
↓
AI crawler fetches it
↓
AI generates an answer directly in the chat interface
↓
User's question is answered, no click required
Cloudflare’s own numbers from last year’s Pay Per Crawl data make the imbalance concrete: Anthropic’s crawler was fetching roughly 38,000 pages for every one visit it referred back to a publisher, and OpenAI’s ratio sat around 1,091 crawls per referral. Publishers were feeding the pipeline and getting almost nothing in return.
Layer on top of that a second, less discussed problem: waste. Cloudflare found that more than half of all AI crawler traffic is spent re-fetching pages that haven’t changed since the last crawl. That’s bandwidth cost for the publisher and compute cost for the AI company, spent on nothing. Cloudflare CEO Matthew Prince framed the broader shift bluntly: “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.” Bot traffic crossing 50% of all web requests was the threshold that made this a “do it now” problem rather than a “someday” problem.
Pay Per Use: the second half of the announcement
Blocking is the stick. Cloudflare’s other move, launched the same day, is the beginning of a carrot.
The company’s original Pay Per Crawl product, launched in 2025, let publishers charge AI companies per fetch, using the long-dormant HTTP 402 “Payment Required” status code as the plumbing. It worked, but Cloudflare now describes it as the wrong unit of value. A page can get crawled once and cited in ten thousand AI answers, or crawled a thousand times and cited in none. Charging per crawl rewards the wrong behavior.
The successor is called Pay Per Use, and the shift is from paying for the fetch to paying for the outcome, specifically whether a piece of content actually ends up inside an answer a user reads. Two partners are live at launch:
- Ceramic.ai runs a pay-per-query model. Publishers who opt in get paid each time their content surfaces in a Ceramic AI search result, and in return get reporting on which queries triggered the appearance, the exact snippet shown, and their ranking position, a discipline Cloudflare is calling answer engine optimization, the AI-era sibling of SEO.
- You.com lets an AI agent pay on demand for a specific piece of premium content at the moment it’s actually needed, without an upfront licensing deal.
No pricing has been published for either arrangement. Cloudflare is framing itself as infrastructure here, not as the party setting the price. The stated pitch is that it provides the plumbing so publishers and AI companies can build their own commercial terms on top of it, the same way it didn’t set ad rates when it moved into ad infrastructure. Beehiiv is also rolling out dashboards for newsletter creators to track and selectively permit AI access, and Patreon has said its default posture will block known training crawlers while still allowing discovery.
Cloudflare frames the philosophical shift like this, in its own writeup: the old web optimized search to save a human time, ten blue links and a click. In an agentic web, an agent can read fast and query continuously, so search stops being a destination and starts behaving more like a utility a system calls dozens of times to answer one question. If that’s the frame you accept, “pay per crawl” was always measuring the wrong thing.
How a bot proves what it says it is
None of this works if crawlers can just lie about which category they fall into. Cloudflare’s answer is Web Bot Auth, a cryptographic signing scheme built on IETF RFC 9421 (HTTP Message Signatures).
The mechanics: an AI company generates a signing keypair, publishes the public key at a well-known URL on its own domain, and signs every outbound crawler request with the private key. The receiving server, or Cloudflare acting as reverse proxy, checks the signature against the published key and gets a cryptographically backed answer to “is this really who it claims to be.” That’s a real upgrade from the status quo, where bot identity is usually just a User-Agent string, trivially spoofed, or an IP range, constantly changing.
Cloudflare is also drawing a distinction between two kinds of automated traffic that get treated differently:
- Verified bots act on behalf of one operator doing one repeatable job. A classic AI training crawler like GPTBot fetching and indexing pages fits here.
- Signed agents act on behalf of individual end users doing one-off tasks. An agent booking a flight for a specific person, often through a remote browser product like Browserbase or Anchor Browser, fits here instead.
The distinction matters because a verified bot and a signed agent should probably be governed by different rules, since one represents a company’s blanket intent and the other represents a single human’s specific request in the moment. OpenAI has already started signing its Operator traffic this way, and Anthropic’s Claude, Perplexity, and several Google crawlers are in the current supported set. Google’s full migration, given the sprawl of Googlebot’s existing infrastructure, isn’t expected until later in 2026.
What this means if you’re building with LLMs
A few concrete shifts, not abstractions:
Retrieval-augmented generation just got a permissions layer it didn’t used to need. If your RAG pipeline fetches live web content to ground an answer, that fetch is now an Agent-category request, and a growing number of sites will have that category blocked by default. Build for graceful degradation: know what happens to your pipeline when a fetch returns a 403 instead of a page, and don’t assume the open web stays open.
“Just scrape it” stops being a viable data strategy for training corpora. If you’re assembling training or fine-tuning data from the web, expect an increasing share of it to require either a declared, authenticated Training-category bot or an actual licensing arrangement. Undeclared, unauthenticated bulk collection is exactly the behavior this policy exists to choke off.
Licensing deals stop being a nice-to-have for anyone doing this at scale. Cloudflare’s own count puts more than 50 major publisher-to-AI-platform licensing agreements signed in the last year, and this policy is explicitly designed to accelerate that number, not shrink it.
If you operate a crawler or an agent, declare it honestly and sign it. A mixed-purpose bot that quietly does double duty is the one category this entire framework is built to punish. Running separate, declared, Web Bot Auth–signed bots for search versus training versus live agent use isn’t just good citizenship anymore, it’s the difference between getting through the gate and getting blocked at the ad-supported pages that make up most of the commercially relevant web.
If you run a site and use Cloudflare, this is a decision you now have to make on purpose. The default answer as of September 15 is no. If you want AI referral traffic, or want to participate in Pay Per Use, you have to go into the dashboard and say so. Passive inertia now means invisibility to AI systems, not neutrality.
The open question nobody can answer yet
Cloudflare calls the next ten weeks, before September 15 hits, an “engagement and testing window.” That’s an admission the defaults could still move before launch.
The bigger open question is adoption. This entire framework only functions if AI companies actually split their crawlers by declared purpose and sign their traffic instead of routing around the whole system with generic browser automation that doesn’t identify itself at all. Cloudflare controls the front door for a large share of the web, which gives its defaults real teeth, but it doesn’t control what OpenAI, Anthropic, Google, or the next well-funded agent startup decides to build. Google in particular has shown no public sign of splitting Googlebot into cleanly separated crawlers, and Googlebot is exactly the kind of mixed-use crawler this policy is built to squeeze.
There’s also a structural risk sitting underneath the publisher-friendly framing. If Cloudflare becomes the layer that verifies bot identity, sets the permission defaults, measures usage, and routes the payment, that’s identification, permissioning, metering, and settlement all sitting with one company that already fronts a fifth of the web. Publishers gain real leverage over AI companies in this arrangement. They also become more dependent on a single intermediary to exercise it. Both things are true at once, and which one dominates in practice is not something this announcement settles.
What is settled is the direction. For the first three years of the generative AI boom, the working assumption was that the public web was fair game unless a site went out of its way to say otherwise. September 15, 2026 is the date that assumption flips for a meaningful slice of the internet. The default stops being “AI can take it.” The default becomes “prove you should be let in, and be ready to pay for what you use.”
Sources: Cloudflare, Cloudflare — Pay Per Use, Cloudflare — Web Bot Auth, Cloudflare Developer Docs, TechCrunch, Engadget, Help Net Security, MLQ News
Comments
Loading comments…