Cloudflare and beehiiv add AI crawl controls for newsletter publishers

Cloudflare and beehiiv moved AI search visibility one layer deeper on June 23, putting crawler governance directly inside a newsletter publisher’s workflow. The integration embeds Cloudflare’s AI Crawl Control into beehiiv, so publishers can manage AI bot access from the same platform they use to publish, rather than juggling separate security or analytics tools.

Cloudflare’s AI Crawl Control documentation lets publishers monitor which AI crawlers are hitting their articles, allow crawlers they consider useful, block the rest, and check whether those bots are respecting robots.txt directives. AI traffic is a measurable audience segment, one that can consume pages without sending much back in referral traffic.

Cloudflare puts that figure at thousands of scrapes for every referral. Its broader AI Crawl Control stack gives site owners three basic choices: allow, charge, or block. The pay-per-crawl system lets a domain owner set a flat per-request price for a zone, then return an HTTP 402 Payment Required response when a crawler does not pay. Paying customers can also send customized 402 messages with licensing instructions, turning bot access into a commercial negotiation instead of an invisible drain.

Broader crawler access can help AI search systems discover archives and surface work in new interfaces. Blocking crawlers can protect content for licensing, subscriptions, or other monetization schemes. Beehiiv co-founder and chief executive Tyler Denk said the move gives creators leverage as AI changes how people find and consume content.

Cloudflare introduced AI Crawl Control in 2025, after an earlier AI Audit phase, and moved it toward general availability. Beehiiv users have begun seeing the controls in beta, and beehiiv Max customers can block AI crawlers.