Cloudflare and beehiiv add AI crawl controls for publishers

Cloudflare and beehiiv have embedded AI Crawl Control directly into beehiiv’s publishing platform, giving newsletter publishers one place to see AI bots, allow them, block them or put a price on access. The June 23 announcement shifts the fight over AI search visibility away from pure optimization and toward access control, where publishers decide which machines can ingest their work at all.

Cloudflare’s AI Crawl Control docs say publishers can monitor which AI crawlers are hitting articles, allow crawlers they consider beneficial, block others and track whether those bots are following robots.txt directives. Cloudflare also says AI crawlers may scrape webpages thousands of times for every referral they send back, a reminder that crawler traffic and referral traffic are not the same thing for publishers trying to measure value.

The product line now gives site owners three basic choices: allow, charge or block. Cloudflare’s pay-per-crawl system lets publishers set a flat per-request price by zone and return an HTTP 402 Payment Required response when a crawler does not pay. Paying customers can also send customized 402 messages with licensing instructions, turning crawler access into a direct negotiation instead of an invisible drain on content.

Cloudflare introduced AI Crawl Control in 2025 after an earlier AI Audit phase, and the beehiiv integration pushes that model into a mainstream newsletter workflow. Cloudflare said the partnership was aimed at helping independent publishers navigate the AI era and giving creators clearer visibility and more granular control over how AI models use their work.

Tyler Denk, beehiiv’s co-founder and chief executive, framed the shift in blunt terms: "publishers need real leverage as AI changes how people find and consume content." That is the core business question behind this rollout. Some publishers will want wider crawl access because AI search can surface archives and send readers back. Others will choose tighter blocks to preserve content for licensing, subscriptions or other monetization.

Beehiiv’s rollout also shows where the market is heading. Some features first appeared in beta, and beehiiv Max customers can block AI crawlers outright. For publishers, AI bot traffic is no longer just a server-log nuisance. It is becoming a policy layer inside the publishing stack, with visibility, permissioning and referral economics all tied to the same dashboard.