ChatGPT shopping picks change 80% when search is enabled

ChatGPT’s shopping answers turned out to be far less stable than many merchants would like to believe. In a test of 1,000 product prompts run 10 times with search enabled and 10 times with search disabled, the product recommendations changed 80.2% between the two modes, leaving just 19.8% overlap across 20,000 responses.

That kind of swing matters because it means visibility in ChatGPT is not a single ranking problem. A product can surface often when the model leans on one knowledge path, then fade almost completely when search is switched on and the answer is rebuilt from live retrieval. The study also found that search-enabled responses averaged 5.2 products, compared with 6.2 when search was disabled, so every slot in the answer carried more weight once search entered the mix.

The analysis tried to keep the test clean by standardizing product names so spelling and naming variations would not distort the results. Even with that control in place, the overlap stayed low. The study also found a 0.4 Pearson correlation between cited-source mentions and recommendation frequency, which suggests cited sources may matter, but does not prove they are driving the recommendation itself.

For merchants, the practical problem is volatility. A brand that looks dominant in one mode may be structurally fragile in another, especially if its visibility depends on how ChatGPT is assembling answers from memory, custom instructions, structured third-party data, or model-generated reasoning. Search Engine Land had previously summarized OpenAI documentation saying product results can draw on all of those inputs, along with safety filters, and that mix helps explain why the shortlist can reshuffle so aggressively.

OpenAI has been pushing deeper into shopping for more than a year. On April 28, 2025, it rolled out shopping features in ChatGPT Search for fashion, beauty, home goods, and electronics, and said the results were not ads, with product data coming from structured metadata and data feeds. On March 24, 2026, it said it was expanding product discovery with richer, more visual shopping powered by the Agentic Commerce Protocol, adding side-by-side comparisons, price and review details, and fresher product information for free, Go, Plus, and Pro users.

OpenAI then described shopping research in June 2026 as a deeper mode for comparisons and tradeoffs, especially in electronics, beauty, home and garden, kitchen and appliances, and sports and outdoor. That makes the new 80.2% finding more than a curiosity. It shows that as ChatGPT’s shopping stack grows more capable, it is also becoming less predictable, and merchants chasing AI discovery will need to test, monitor, and re-test across modes if they want any shot at stable conversion.