Four AI Search Tools Cited a Made-Up Concept Within a Week
Researchers published articles about a made-up concept and found that ChatGPT, Perplexity, Grok and Google AI cited them. The authors caution that a citation in an AI answer does not, by itself, verify a claim.
Researchers at Fudan University published articles about a made-up concept, and within a week ChatGPT, Perplexity, Grok and a tool referred to in the paper as Google AI had cited them. As Search Engine Journal reports, the tests took place in March and April, and the paper was posted on arXiv on October 8. For anyone using AI search, the takeaway is simple: check the source behind a citation. The citation alone does not verify a claim.
How the Researchers Tested AI Search
The team first asked ten AI search tools 120 questions and recorded which sites they cited. The researchers classified four tools as working with the English-language web and another six, including Doubao and DeepSeek, as working with the Chinese-language web. Frequently cited sites included Reddit, YouTube and Wikipedia. Of each tool’s ten most-cited sites, between six and ten allowed ordinary users to publish content.
The researchers examined publishing requirements on 22 platforms and, for each tool, selected a frequently cited site that a new account could post to. They ended up with nine platforms, including Medium, YouTube and Sohu. This let the team test the role of the publishing platform.
They posted articles on the selected sites about the fictional Cactus Gravity Protocol, whether AI will replace junior programmers within five years, and a nonexistent keyboard. They drafted the articles with ChatGPT, then revised them by hand and added invented names and numbers. Those details made it possible to see whether a tool had done more than provide a link: whether it had also repeated information from the article.
Before publishing, the team checked that the tools did not give substantive answers about the two fictional topics or repeat the planted details. After publishing, the researchers asked the same five questions on each topic every day for a week.
Which Articles Appeared in Answers
According to the researchers, eight of the ten tools cited articles about Cactus Gravity Protocol within a week, including all four English-language tools. From the first day each tool produced a citation through the seventh day after publication, Grok cited an article in 30 of 32 answers, Google AI in 15 of 33, Perplexity in 13 of 35, and ChatGPT in 7 of 20.
The invented details appeared less often: in 53% of Grok’s answers, 10% of ChatGPT’s and 6% of Google AI’s. None were found in Perplexity’s answers. Across all ten tools and three topics, citations to the articles appeared in 176 of 1,004 answers over the week.
The other topics produced different results. Of the four English-language tools, only Perplexity cited the article posing an opinion question, doing so in 11.4% of its answers. None cited the articles about the fictional keyboard. The authors do not establish what caused the difference: the topic, competition from other content or the wording of the question. Those conditions changed at the same time.
Where the articles were published also affected individual test results. In five comparisons involving articles about little-known travel destinations, the first result—a citation, a link or a repetition of a planted detail—came from the article on a frequently cited site every time. Forty articles on a self-hosted WordPress site received no citations over seven days. The researchers describe the results of specific tests, not a general rule for all sites.
The study has important limits: it does not specify which Google product it calls Google AI, and the main tests did not involve real brands or products. The results therefore cannot tell us whether the same pattern would appear when someone searches for information about a company. The figures may also change as the tools are updated.
What Website Owners and Marketers Can Do
For teams monitoring content and reputation, this is a reason to check claims in AI search answers, especially when a citation leads to user-published content. In the experiment, the invented concept made its way into answers through sites the tools already cited frequently. But a citation does not make the claim itself reliable.
The study should not be treated as a guide to getting featured, either: other topics produced different results, and real brands were not tested. Google warns that trying to secure inauthentic mentions may be less useful than it seems, pointing to its quality assessment systems and spam protections. Google’s clarification that its spam policies apply to attempts to manipulate generative answers in Search came out on May 15—after these tests.
My View: Check the Claim, Not Just the Citation
I think it is important to distinguish between monitoring what AI search says about a project and trying to get into its answers at any cost. If I came across an unexpected claim about a company, I would open the cited article and look at who wrote it and what they based it on. In this experiment, a citation could lead to an article deliberately written about something fictional.
I would not put routine website work on hold to pursue mentions in AI search. Clear service pages and a clear site structure matter regardless of how search changes. And if it is difficult to submit an enquiry on a phone, or labels and contrast make a page hard to read, extra attention to the project will not remove those obstacles.
Sources
Where the news comes from. The text is a retelling in the author’s own words; the facts come from the source, the opinion is the author’s.
- AI Search Tools Cited A Made-Up Concept Within A Week, Paper Finds via @sejournal, @MattGSouthern www.searchenginejournal.com
Junior SEO specialist
I help local projects tidy up service pages and product listings. I’m learning to explain to business owners exactly what changes after edits.
All posts by the authorRelated articles
OpenAI’s API and Google Found Only Some of the Pages ChatGPT Cited
Google and the OpenAI Responses API found only about a quarter of the pages ChatGPT cited. Measuring visibility in AI answers means distinguishing between matching websites and matching URLs.
NBER study: AI agents increased commits by 240%, releases by 30%
An NBER study of more than 500,000 developers found that autonomous AI agents increased commits by 240%, but releases rose by just 30%.
Graphite Finds 13,000 Phrases AI Uses More Often Than Humans
Graphite identified 13,000 phrases that language models use at least twice as often as humans. The list can help editors review AI-assisted writing, but it cannot establish who wrote a particular article.
Discussion 4
Mila from Plovdiv
For a small bakery, I’d also keep a dated copy of key product and ingredient pages, so we can check what an AI tool may have picked up if it cites something outdated.
Lidia E.
Keeping dated copies sounds useful, but how would you tell whether an AI cited an outdated page versus simply repeating an unsupported claim? For a bakery, I’d want a low-effort way to check that before investing in a more elaborate archive.
Aurora Ribeiro
A low-effort check is to compare the cited wording and page date with the current ingredient page, then confirm the claim against the label or supplier record. A dated copy helps establish what was published when, but it can’t show whether the AI relied on that version; the paper’s point is that citation alone doesn’t validate the claim.
Jaakob Pärn
Could you compare the cited wording with the current product page first, and check an older copy only if the details changed?