Cloudflare is changing how websites deal with AI crawlers, and honestly, this was coming.
For years, website owners have watched bots move through their pages, collect content, index it, train on it, summarize it, reuse it, and sometimes send very little traffic back. Search engines were one thing. AI crawlers are something else. The line between discovery and extraction has become messy.
Now Cloudflare is giving site owners a more detailed way to decide what kind of automated traffic gets in. The company has introduced new AI crawler controls that classify bots into three categories: Search, Agent, and Training. The feature is available to all Cloudflare customers, including those on the Free plan.
A Bigger Shift Than Just Blocking Bots
This is not just a new button that says “allow” or “block.”
Cloudflare is trying to separate bots by what they actually do. Search crawlers are meant to help people find pages. Training crawlers are tied to model development. Agent crawlers act on behalf of users or systems. That difference matters because many publishers and website operators do not want to shut out every bot. They still need visibility in search. They still want referral traffic. What they do not want is unlimited content use with no choice, no clarity, and no compensation.
That is where Cloudflare’s new setup becomes interesting. Website owners can apply different rules depending on the crawler’s purpose, instead of treating every AI-related bot as the same problem.
September 2026 Brings a Major Default Change
The biggest date here is September 15, 2026.
Starting on that date, Cloudflare says new domains will block Training and Agent crawlers by default on pages that show ads. Search crawlers will remain allowed by default. That is a pretty direct signal. Cloudflare is not saying all bots are bad. It is saying that crawling for search and crawling for AI training should not automatically receive the same treatment.
There is one catch website owners need to understand. Some crawlers serve more than one purpose. Googlebot, Applebot, and BingBot may be evaluated under multiple categories when they perform both search and training functions. If a site blocks Training crawlers, those multi-purpose crawlers may also be blocked under Cloudflare’s policy, even if Search crawlers are generally allowed.
That could create real decisions for publishers. More control sounds good, but control also means responsibility. A bad setting could affect discoverability. A loose setting could allow more content use than a publisher wants.
Why This Matters for Website Owners
The web is being renegotiated in real time.
For a long time, the basic trade was simple enough: search engines crawled websites, indexed pages, and sent traffic back. AI systems changed the bargain. They can absorb content, summarize it, answer user questions directly, and reduce the need to click through to the original source.
That is why publishers, bloggers, media companies, ecommerce sites, SaaS documentation teams, and independent creators are paying closer attention to bot traffic. It is no longer just a security issue. It is a business issue.
Cloudflare’s AI crawler controls give website owners a way to say: yes to search, no to training, maybe to agents. Or the opposite, depending on their strategy.
BotBase Adds More Visibility for Enterprise Users
Cloudflare is also expanding BotBase, a searchable database of known bots. For Enterprise Bot Management customers, BotBase gives administrators a clearer view of bot classifications and behaviors.
Admins can search for specific bots, view their classification, filter traffic, and copy detection IDs for use in security rules. That sounds technical, but the practical value is simple: site owners get a better idea of who is crawling them and why.
Visibility matters here. Blocking unknown traffic is one thing. Making decisions based on identified bots is far more useful.
Content Use Controls Go Beyond Crawling
Cloudflare is also adding content use controls for Enterprise Bot Management customers. These controls let website owners express how crawlers may use content after accessing it.
The three levels are Immediate, Reference, and Full. Immediate means no storage or reuse. Reference allows indexing, excerpts, and links back. Full allows summaries or reproduction. Cloudflare is extending its Content Signals format in robots.txt with a new “use” parameter to express these preferences.
There is an important limitation. Robots.txt does not technically enforce the setting. Still, Cloudflare says it will report whether Verified Bots comply with declared preferences through BotBase. Verified Bots that ignore those preferences or reproduce content in full could lose their Verified status.
That is not perfect enforcement. But it is pressure. And pressure matters when the web is trying to create new norms around AI content use.
Verified Bots Will No Longer Mean Automatic Access
Cloudflare is also changing how verified bots are treated.
Previously, Verified Bots were allowed by default. Under the new model, verification only confirms identity. It does not automatically grant access. Whether a bot can crawl a site will depend on its classification and the website owner’s policies.
That is a major philosophical change. Being known is no longer the same as being welcome.
AI Agents Create a New Trust Problem
Cloudflare also proposed a transitive trust model for AI agents and bots that operate through third-party platforms. The idea is to identify the original requester behind an intermediary using the standard HTTP Forwarded header.
In plain terms, if an AI agent reaches a website through another service, the website owner should still be able to know who is really behind the request. Cloudflare said this could help site owners apply access policies based on the original bot operator, not just the platform passing along the traffic.
That is useful, but not simple. Privacy, anonymous access, and agent-based browsing will all make this harder than a clean policy diagram suggests.
The Web Is Getting More Defensive About AI
Cloudflare’s new AI crawler controls show where the internet is heading. Website owners are no longer comfortable leaving content access on autopilot.
Search traffic still matters. AI discovery may matter too. But training data, summaries, automated agents, and content reuse are now part of the same conversation.
The old web assumed crawling was mostly good. The new web is asking a different question: crawling for what?
And that question is not going away.