AWS, Cloudflare race to build infrastructure for AI agents

The internet’s busiest traffic is no longer just human. AWS, Cloudflare, Microsoft and Google Cloud are redesigning cloud infrastructure for a future in which AI agents, not people, generate more of the web’s requests, and that shift is already changing who controls access, who pays for the load, and how open the network remains.

AWS says it is helping customers move beyond experiments to “production-ready agentic AI systems” and is shifting enterprises “from model deployment to agent infrastructure.” Microsoft described the coming internet as an “open agentic web” in its May 19, 2025 Build announcement, while Google Cloud has been marketing “agent-ready systems” and a dedicated enterprise agent platform. Taken together, the message from the major cloud providers is clear: AI agents are moving from demos into production, and the underlying plumbing of the web is being rebuilt to keep up.

Cloudflare’s traffic data shows why the race matters. In 2025, Googlebot was again the highest-volume request source to Cloudflare. Crawler traffic rose 18% from May 2024 to May 2025, while GPTBot traffic grew 305% over the same stretch. Cloudflare also said 14% of top domains now use robots.txt rules to manage AI and search crawlers, a sign that publishers and platform owners are already drawing lines around automated access as machine traffic becomes more common.

The scale of the infrastructure shift is not abstract. Cloudflare says its network spans 330 cities in more than 125 countries and regions, and it handles more than 81 million HTTP requests per second on average, with more than 129 million at peak. That global footprint puts the company at the center of a change with public-interest consequences: as more of the internet is mediated by bots, the cost of serving content, the rules for scraping it and the ability to block or charge for access all move upward into the hands of a few infrastructure providers.

Google Cloud’s 2025 State of AI Infrastructure report underscores how quickly enterprise adoption is advancing. It says 98% of organizations are exploring generative AI and 39% are already deploying it in production. Cloudflare’s year-end review adds another marker of change, saying human-generated web traffic that is post-quantum encrypted reached 52%. The pattern is the same across the stack: the web is being tuned for automated systems first, and ordinary users may end up on a slower, more gated internet shaped by the companies that can see, filter and monetize the machines.