Cloudflare gives publishers new controls over AI bot access

Cloudflare moved on July 1 to give website owners a more precise way to decide how AI systems use their material, replacing a single “Block AI bots” switch with behavior-based controls for Search, Agent and Training traffic. Starting September 15, 2026, new domains onboarding to Cloudflare will block Training and Agent bots on pages that display ads by default while leaving Search allowed, and customers can opt out before then.

The update sharpened a problem publishers have been trying to solve since AI crawlers became a bigger part of the web economy: not every automated visitor behaves like a search engine, and not every visit brings value back. Cloudflare’s earlier 2025 tools already let site owners manage robots.txt on their behalf and block AI bots only on pages monetized through ads. Its pay per crawl beta, introduced in 2025, went a step further by letting content owners charge AI crawlers for access. The company’s position is that the old bargain, where search indexing sent traffic back to original publishers, has been weakened as AI systems increasingly consume derivatives without returning the same audience.

That distinction matters inside monday.com, where more than 250,000 customers worldwide use the platform and where content is part of the product machine. Product pages, help centers, research reports and blog posts are not just marketing assets; they are discovery surfaces, support channels and sales proof points. Cloudflare’s new defaults point to a harder operational choice for teams that build that material: let automation index and surface it, or lock down access when the same content can be scraped for training or used to answer questions elsewhere without sending visitors back.

The issue also reaches beyond marketing. Security and legal teams are now being asked to think about bot governance the way they already think about permissions, compliance and analytics. monday.com has been leaning into its own AI story, describing itself as an AI work platform, saying AI agents execute work on the platform, and using Sidekick in workdocs to generate text inside documents. That makes Cloudflare’s classification scheme relevant to product teams as well, because customers will increasingly expect transparent controls over what AI can touch and what it can do once it gets there.

For SaaS companies built on trust, the new Cloudflare model makes the web look less like an open buffet and more like a set of negotiated terms. Search stays welcome on ad-supported pages, but training and agent access now come with explicit rules, and Cloudflare is betting that publishers will want the right to decide which kind of machine gets in.