Google expands Search Console insights for AI search visibility

Google is adding Search Console reports that show impressions, the URLs that appeared in generative AI features and the countries where those appearances occurred, then rolling those reports out first to a subset of websites. The move puts AI search visibility on the same operational footing as classic Search reporting: if a page cannot be crawled, refreshed or legally used, it cannot surface in the answer layer.

That plumbing now spans more than one system. Google says web publishers have long used open standards such as robots.txt to control how content appears in Search, and the same control logic now reaches Search AI features. Google-Extended is the company’s robots.txt-based setting for deciding whether a site’s content can help improve AI models, while Google says it is also testing new controls for how links and content appear in generative AI Search features. AI Overviews, which Google said had expanded to more than 200 countries and territories and more than 40 languages by May 2025, are already part of the mainstream Search experience.

Discovery still starts with basic feed discipline. Google Search Central recommends XML sitemaps and RSS, mRSS or Atom 1.0 feeds together, since sitemaps describe the full URL set and feeds communicate recent changes. A single XML sitemap is capped at 50MB uncompressed or 50,000 URLs, so larger sites need sitemap index files to point to multiple sitemaps. Google says Search Console can accept up to 500 sitemap index files per site account, which makes the index layer a practical control point for large publishers and multi-market properties.

The same crawl logic is increasingly shared across AI systems. OpenAI says OAI-SearchBot is the crawler used to surface websites in ChatGPT search answers, while GPTBot is a separate control for training data, and the two settings are independent. OpenAI says crawling stops immediately when robots.txt disallows access, although a disallowed page can still surface as a link and title in ChatGPT Atlas if it is discovered through a third-party search provider or another crawl path unless it is noindexed. Anthropic says its general-purpose web crawler follows industry-standard robots.txt instructions.

For publishers, the order of operations is now clear: check robots.txt, confirm noindex and feed freshness, then use Search Console to see where AI features are already surfacing pages and where they are missing. The visibility problem often starts long before ranking does.