Google tests Web Bot Auth to verify legitimate crawlers

Google has started testing Web Bot Auth on some AI agents hosted on Google infrastructure, and the experiment goes straight at a problem publishers know too well: figuring out which automated visitors are real and which ones are bluffing. In Google’s documentation, a subset of requests made by Google-Agent were signed with Web Bot Auth, while the company said it was not signing every request and still advised site owners to lean on IP addresses, reverse DNS, and user-agent strings during the rollout.

That matters because Web Bot Auth is not just another detection trick, it is a move toward cryptographic proof of identity for crawlers and agents. Google described the protocol as a draft from the IETF WBA Working Group, which means it can still change, but the direction is clear. Instead of trusting a bot because it says who it is, site operators may increasingly expect it to prove it. For publishers, that could shift crawl policy from a guessing game to a rules-based system built around authenticated traffic, with direct consequences for visibility, access, analytics, and content control.

The IETF’s working group is trying to standardize exactly that layer. Its charter covers crawlers for search indices, web archivers, link checkers, crawlers for AI training, and AI agents that retrieve or interact with content on behalf of end users. It also says older verification methods, including IP allowlisting, user-agent strings, and shared API keys, have major limits in security, scalability, and manageability. That is a strong signal that the next fight over scraping will not be about whether bots exist, but about which bots get recognized as legitimate.

Cloudflare’s documentation points in the same direction. The company says Web Bot Auth uses cryptographic signatures in HTTP messages and relies on two active IETF drafts, one for a directory of public keys and one for the protocol itself. Cloudflare’s earlier 2025 work also captured the industry’s new reality: the rise of AI agents has blurred the line between ordinary crawlers and user-directed agents. If those signatures catch on, the practical result could be a new permissions layer for the web, one that helps search engines, publishers, and platforms decide who may scrape, under what rules, and with what proof attached.