Anthropic clarifies Claude visibility across training, search and retrieval bots
Anthropic now splits Claude into separate training, search and retrieval bots, and a June analysis found 39 of 107 sites blocked at least one of them.
Anthropic’s own help documentation makes one thing plain: Claude visibility is not a single switch. The company says it uses different robots for model development, web search and user-directed web retrieval, and frames them as a way to give site owners transparency and choice.
That split matters because each bot serves a different function. ClaudeBot is tied to model development, while Claude-User and Claude-SearchBot map to search and live retrieval behavior. A page blocked from one crawl path may still be reachable through another, which means publishers cannot treat one robots.txt rule as a blanket control for all Claude activity. Anthropic’s help article says site owners can set preferences for each crawler, reinforcing that the company is exposing multiple access surfaces rather than one general-purpose scraper.
The change landed as a visible SEO issue in late February 2026. Search Engine Land published its coverage on Feb. 25, 2026, and Barry Schwartz at Search Engine Roundtable wrote the same day about Anthropic updating its crawler docs to separate ClaudeBot, Claude-User and SearchBot. Search Engine Journal followed on Feb. 26 with a post on Anthropic separating Claude bots for training, search and user requests. The documentation itself is available through Anthropic’s Claude Help Center and Claude Privacy Center, with a robots.txt file on Anthropic’s main site for site owners to inspect.
Google’s own robots.txt guidance helps explain why this distinction matters. Google Search Central says robots.txt is mainly for controlling crawling load, not for keeping a page out of Google entirely. Anthropic’s split pushes that same old distinction into AI search visibility: crawl access, search retrieval and model training are not the same thing, even when they originate from the same company.
The numbers already show site owners are reacting. A June 13, 2026 analysis by US Tech Automations found that 39 of 107 prominent sites, or 36.4%, blocked at least one Anthropic crawler in robots.txt. That is a clear sign that publishers are starting to manage Claude access at the bot level, not as a single policy decision.
The audit questions are now more specific. Block the training bot if you do not want pages used for model development. Check whether search and user-request bots are still allowed if you want Claude to cite or fetch your content. Review access logs to see which Anthropic user agents are actually hitting your pages. Anthropic’s documentation makes the operational point for publishers and SEO teams: visibility in Claude is about which pathway can reach the page, not whether Claude exists at all.
This article was produced by Prism’s automated news system from verified source data, official records, and press releases, then run through automated quality and moderation checks before publishing. The system is built and supervised by the people who set the standards it runs under. Read our full AI policy.
Did this article answer your question?
