AI bots are driving up server costs for publishers

Publishers are being asked to pay the bill for AI systems that often give nothing back. The choice is no longer theoretical: block the bots, slow them down, charge them, or leave the door open and keep subsidizing crawl traffic that can stress servers while cutting into the traffic that once paid the rent.

Cloudflare sharpened that debate on July 1, 2025, when it said customers would default to blocking AI crawlers unless site owners choose otherwise. It also rolled out Pay per Crawl, a setup that records billing when an authenticated crawler request gets a 200-level response with a crawler-charged header, with 402 available when access is paid. That matters because Cloudflare said training drove nearly 80% of AI crawling by mid-2025, up from 72% a year earlier, while GPTBot and ClaudeBot kept growing even as referrals back to publishers fell.

The pressure is not abstract. Fastly’s Q2 2025 Threat Insights Report said AI crawlers made up almost 80% of the AI bot traffic it observed, and one fetcher was hammering a site at 39,000 requests per minute. For publishers running heavy archives, local listings, or image-rich pages, that kind of load is not a rounding error. It is a direct infrastructure cost, and it lands before a single click, lead, or subscription sale comes back.

That is why the strategic question has changed from “Should crawlers be allowed?” to “Which crawlers deserve access?” OpenAI’s documentation separates GPTBot and OAI-SearchBot as distinct user agents that webmasters can manage independently in robots.txt, which gives publishers at least some control over whether they are feeding training systems, search support, or user-facing retrieval. Google’s own crawl budget guidance says site owners can reduce crawl rate if Googlebot overloads a server, but that guidance is about indexing and discovery, not model training. An IETF Internet-Draft published in November 2024 tried to sharpen that distinction further by proposing robots.txt updates for AI-specific crawlers and for separating training from inference.

The economics are getting worse at the same time. Reuters Institute reporting in 2026 said media leaders expected search referrals to fall by 43% over the next three years, and Chartbeat data cited in that reporting showed Google search referrals to publishers fell globally by a third in the year to November 2025. In that market, every crawl that burns bandwidth without sending readers back looks less like normal web traffic and more like unpaid support for the AI stack.