OpenAI’s API and Google Found Only Some of the Pages ChatGPT Cited

Researchers from the National University of Singapore and Xi’an Jiaotong University compared sources cited in ChatGPT answers with results from Google and the OpenAI Responses API. As Search Engine Journal reports, the preprint data was published on October 11, 2026. For GPT-5.5 Instant, the two methods combined found 26.2% of cited pages but 48.7% of cited websites. That distinction matters when measuring visibility in AI answers.

How the sources were compared

The researchers observed the ChatGPT web app in their own logged-in sessions. They recorded the data stream as answers were generated, including the app’s search queries, the pages it retrieved, and the sources it selected. They counted both individual links and pages within source groups.

They tested 80 questions from Search Arena, a dataset of real user queries submitted to chatbots with search. Forty questions were in English and 40 in Chinese. They covered recommendations, fact-finding, information synthesis, and analysis. Each question was run 20 times on GPT-5.3 Instant and GPT-5.5 Instant. There were fewer runs for GPT-5.2 Instant and GPT-5.4 Thinking because those versions were retired before data collection was complete.

As proxies for checking ChatGPT directly, the researchers used Google organic search results via SerpAPI and OpenAI’s Responses API with web search. They collected up to five pages of Google results per question, averaging about 43 results per question. They queried gpt-5.3-chat-latest and gpt-5.5 through the API 20 times each. Those results were collected within three days of running the same question in ChatGPT. The main data collection ran from March 29 to July 29, 2026. The paper is available on arXiv as a preprint and has not been peer-reviewed.

Matching a website is not the same as matching a page

For GPT-5.3 Instant, Google and the API together found 28.3% of cited pages and 50.3% of cited websites. For GPT-5.5 Instant, the figures were 26.2% and 48.7%, respectively. Subdomains were counted as separate websites. The figures were averaged across the 80 questions; a page counted as cited if ChatGPT selected it as a source in at least one of the 20 runs.

On its own, Google found 9.4% of pages cited by GPT-5.3 Instant and 6.4% of pages cited by GPT-5.5 Instant. At the website level, the shares were 26.1% and 22.8%. The API’s full source list covered more pages: 22.9% and 22.5%. But the pages the API cited in its own answers were only a subset of that source list.

The same question did not produce a consistent set of pages. Of the pages that appeared in GPT-5.5 Instant results for a given question, 44.5% appeared in just one of the 20 runs. For GPT-5.3 Instant, the figure was 40.8%. Websites overlapped more often across runs. When search activity was visible, neither version used the question’s original wording verbatim. For questions in Chinese, visible searches were conducted only in English in 67.0% and 69.6% of runs, respectively.

What this means for visibility checks

In these tests, Google results and the Responses API did not reproduce the exact list of pages ChatGPT cited. Matches were more common at the website level than at the individual URL level. So measuring visibility calls for two separate questions: does the website appear among the sources, and does the intended landing page appear?

The authors recommend checking both levels, running the same question multiple times, and accounting for how ChatGPT turns a user’s wording into its own search queries. When choosing a tracking method, it is important to establish whether the data comes from an API or directly from the ChatGPT app. The researchers did not test commercial tools, so this study cannot establish the accuracy of any particular service.

All the configurations studied predate GPT-5.6 and GPT-6. According to OpenAI’s help page, GPT-6 is rolling out to eligible Plus, Pro, Business, and Enterprise users, while GPT-5.6 Luna is becoming the default option for Free and Go. The percentages in the preprint describe the versions studied, not the newer models. The authors recommend revisiting the monitoring approach when the model changes.

My take: checking whether a website appears is not enough

I wouldn’t infer the visibility of a specific page from a single domain-level report. For an SEO task, the first step is to identify user intent and check whether the landing page meets it. Only then would I compare that page with ChatGPT’s sources. A website appearing among the citations does not confirm that the user saw the content the team wanted them to see.

An additional monitoring system is justified if it is clear what decision the editorial or product team will make based on its data. I would start by checking how sources are collected and running the question several times. Otherwise, it is easy to mistake the difference between a website and a page for a change in SEO performance.

Sources

Where the news comes from. The text is a retelling in the author’s own words; the facts come from the source, the opinion is the author’s.

1. OpenAI’s API Only Partly Matched ChatGPT’s Citations, Paper Finds via @sejournal, @MattGSouthern www.searchenginejournal.com

About the author

Úrsula Gomes

Web project analyst

I gather requirements, review funnels, and help teams keep business goals in sight amid redesign tasks. For an SEO review, I usually start with search queries and site analytics.

All posts by the author