Ahrefs study finds llms.txt files get almost no traffic, despite hype

Most llms.txt efforts look like performative compliance, not a visibility strategy. Ahrefs analyzed 137,000 domains with its Web Analytics and Bot Analytics and found that 28% of the sites had an llms.txt file, yet 97% of those files received zero traffic in May 2026. The file that many marketers treated like a fast lane into AI answers was, in practice, mostly ignored.

The few requests that did land were overwhelmingly automated. Ahrefs said 96% of the traffic to llms.txt files came from bots, and named AI tools accounted for only 19.5% of fetches among the small subset of files that were actually touched. GPTBot and Claude-Code were the biggest named requesters in the dataset, while much of the rest of the activity came from tools auditing or scanning the files rather than from mainstream AI search systems. For site owners hoping a simple root-level markdown file would move the needle, that is a hard return on effort to justify.

That matters because llms.txt has been sold, in some corners of the SEO world, as a cleaner way to steer language models toward the right pages and away from junk. Jeremy Howard of Answer.AI published the proposal on September 3, 2024, as a Markdown file meant to summarize a site and point models to key resources. But the proposal does not define a processing method, and Google’s own guidance now says special machine-readable files are not needed to appear in generative AI search. Google says its AI Overviews and AI Mode lean on core ranking and quality systems, along with retrieval-augmented generation and query fan-out, not on a new compliance file.

The split got sharper in late May, when Chrome’s experimental agentic browsing audits added an llms.txt check even as Google Search guidance said the file was unnecessary for visibility. Google’s December 8, 2025 security note on agentic browsing also stressed the risk of indirect prompt injection and said Chrome’s work in this area relies on layered defenses and user confirmations. That leaves llms.txt in an awkward middle ground: useful as a signal some browser agents may inspect, but not a lever that changes how Google Search surfaces pages in AI features.

For AI search teams, the Ahrefs numbers point to a cleaner priority list. Watch real crawler behavior, strengthen content architecture, and make sure structured signals are doing their job. If the goal is crawl, citation, and inclusion in AI answers, the evidence says the file alone is not the shortcut many hoped it would be.