AI bots are driving up site costs, not just scraping content

AI bots are starting to look less like a content problem and more like a line item. The newest shift is not simply that machines are visiting sites more often, but that they are consuming bandwidth, triggering server load, and creating costs without always creating a single human visit in return. For agencies, that turns technical SEO into something closer to infrastructure consulting, where the real value is helping clients protect margins while keeping their sites discoverable.

The bill is now part of the traffic

The clearest warning sign is the scale of machine activity. Akamai Technologies, based in Cambridge, Massachusetts, said AI bot traffic skyrocketed 300 percent in one year in its November 4, 2025 State of the Internet report. Cloudflare reported in early June 2026 that automated systems accounted for 57.4 percent of requests, compared with 42.6 percent from humans, and CEO Matthew Prince said the crossover happened faster than he expected. Those numbers matter because they show AI bots are no longer a side issue in web operations, they are the dominant traffic class on many properties.

This is the part that changes the agency brief. A site can now be “successful” in visibility terms and still absorb a real operational penalty from machine activity. If bots are using the site as a resource, then hosting, caching, log analysis, and request management become as strategic as rankings and clicks.

What the major reports say

Akamai’s report made the publisher angle impossible to ignore. Publishing organizations accounted for 40 percent of all OpenAI requests, while AI training crawlers made up 63 percent of AI bot activity targeting media and AI fetchers made up 24 percent. Akamai returned to the theme in an April 8, 2026 follow-up focused on publishing, saying the sector was under attack and noting that commerce logged more than 25 billion bot requests in a two-month span.

Other security firms are describing the same acceleration from different angles. HUMAN Security said monthly AI-driven traffic nearly tripled during 2025, AI agent traffic grew 7,851 percent, and post-login account compromise attempts more than quadrupled year over year. Thales added that AI-driven bot attacks surged 12.5x in 2025, bots made up 53 percent of observed internet traffic, malicious bot activity represented 40 percent of total internet traffic, and daily blocked requests rose from 2 million to 25 million.

Taken together, those figures do more than confirm growth. They show that bot activity is now large enough to affect bandwidth bills, infrastructure planning, abuse prevention, and revenue forecasts all at once.

The behavior that makes it expensive

The Search Engine Journal reporting by Roger Montti and Danny Goodwin adds a practical example that makes the cost problem easy to picture. The article described a recurring pattern in which Meta’s meta-externalagent crawler followed URL variations for days before mitigation systems caught the behavior. That kind of looping activity is exactly what turns automated traffic from a theoretical concern into a measurable expense, because it can generate request volume without producing any useful business outcome.

That distinction matters. A bot that crawls efficiently may still be unwelcome, but a bot that gets stuck in an inefficient loop is a direct drain on servers, logs, and response capacity. It is not just taking content, it is taking compute.

Why agencies are being pulled into a new service category

This is where the editorial shift becomes obvious. The work is moving beyond crawl policy into bot governance, and that means agencies need to talk to devops and security teams as often as they talk to content strategists. The most useful advice may now include server log analysis, selective blocking, rate limiting, caching strategy, and tighter robots policy enforcement.

That creates a new consulting offer with a very practical pitch: protect the site’s infrastructure while preserving discoverability. Agencies that can help clients balance access with control are not just solving a technical nuisance, they are helping justify marketing spend by preventing hidden hosting costs from eating into the budget. In that model, technical SEO becomes an operating-cost discipline as much as a traffic discipline.

The business argument is straightforward. If machine traffic is rising faster than human traffic, then the value of visibility depends on whether the site can absorb the load without degrading performance or inflating costs. The agency that can prove it understands both sides of that equation becomes much more strategic than the one that only reports keyword movement.

What bot governance actually covers

Bot governance is not one tactic. It is a stack of decisions that has to be managed together.

• Server log analysis to identify abnormal request patterns and looping crawls
• Crawl policy tuning to define what should and should not be fetched
• Robots rules that reflect current site priorities rather than old defaults
• Rate limits and selective blocking to stop abusive or low-value behavior
• Caching and edge controls to reduce the cost of repeat requests
• Coordination with security teams when bots shift from nuisance to fraud risk

That toolkit is becoming more urgent because the industry conversation has changed. The old debate centered on scraping and content ownership. The new debate is about measurable cost exposure, request volume, and whether a site is effectively paying to be consumed by machines.

Why publishers and commerce feel it first

Akamai’s numbers make clear why publishers are on the front line, but commerce is not far behind. Content-rich businesses have always been attractive to crawlers, yet the current wave of AI traffic increases the stakes because every request can carry both operational and commercial consequences. A publisher may lose margin to crawling overhead, while an ecommerce site may face that same overhead alongside fraud pressure and post-login abuse.

That overlap is why security teams are now treating AI traffic as both an infrastructure issue and a fraud issue. HUMAN Security’s spike in account compromise attempts and Thales’ jump in blocked requests show that machine traffic is not only about reading pages. It can also be part of broader abuse patterns that force companies to harden systems that were never designed for this volume of automated pressure.

The strategic takeaway for agencies

The big shift is not that bots exist. It is that AI bots are now active enough to threaten the economics of being visible on the web. Cloudflare’s June 2026 crossover point, Akamai’s 300 percent surge, HUMAN Security’s near-tripling of monthly AI-driven traffic, and Thales’ 12.5x rise in AI-driven attacks all point in the same direction: machine traffic is becoming a core part of web operations, not an edge case.

That is why the smartest agency play is changing. The most valuable retainers will not just promise clicks or rankings. They will help clients defend infrastructure, control crawl behavior, and explain why some of the new cost of doing business online is no longer human at all.