Ahrefs study finds llms.txt is barely used by AI tools

Agencies do not need to sell llms.txt as a must-have SEO fix. Ahrefs logged 137,000 domains and found that 28 percent had published the file, but 97 percent of those files received zero traffic in May 2026. Even among the small slice that was fetched, 96 percent of requests came from bots rather than humans, and only 19.5 percent of fetches came from named AI tools.

That should change how account managers talk to clients. llms.txt was proposed by Jeremy Howard and published on Sept. 3, 2024 as a way to give large language models a concise Markdown summary of a website plus links to deeper pages. In theory, it is a neat shortcut for machine readers. In practice, Ahrefs’ log data shows that most AI systems are still not bothering to use it at scale, which makes it a weak candidate for the top of any agency checklist.

The rest of the ecosystem makes the same point in a different way. Google’s May 2026 guidance for generative AI search says site owners do not need new machine-readable files, AI text files, markup, or Markdown to appear in generative AI results, and it specifically names llms.txt in that category. Google also says generative AI features still depend on foundational SEO work, clear technical structure, and useful non-commodity content, which is where budgets still belong.

Chrome is pushing in a different direction, at least experimentally. Chrome for Developers now documents an Agentic Browsing category in Lighthouse that includes an llms.txt audit, with language suggesting agents may spend more time crawling a site if the file is missing. But the tooling is still in its own sandbox, requiring Chrome 150 or later, while WebMCP audits require registration for the WebMCP origin trial. That is a lot of motion around a file that, in real-world traffic, is still mostly invisible.

Ahrefs’ own breakdown underscores how thin the adoption layer is. GPTBot was the most visible named bot in the study, with Claude-Code also showing up prominently, but the larger pattern was that named AI systems were not actively looking for llms.txt files in meaningful volume. For agencies, that means the smart move is to deprioritize llms.txt unless there is a specific workflow that truly needs it, and put the energy into crawlability, internal linking, content depth, and authority signals that search systems and AI crawlers already encounter every day.