Skip to main content

AI search returns deleted pages, not fake URLs

Our write-up of a story first reported by Habr SEO

AI search returns deleted pages, not fake URLs

An author checked around six hundred URLs from five AI models with search about a month after collecting them. Nine out of ten opened. Dead pages with 404 and 410 codes numbered a handful. These are addresses of standard content on live platforms, not random strings, so most likely deleted articles rather than hallucinations. When exactly they were removed is not known.

A hallucinated link is when the model assembles a plausible address for a page that never existed. AI invents sources where it does not search the web: in a Walters and Wilder study without search, 55% of academic links from GPT-3.5 and 18% from GPT-4 were fabricated. Among the URLs that did not open, two groups emerged. Around three dozen did not serve the page to a bot — site protection. Dead ones numbered a handful: agency compilations on vc.ru and DTF and a blog on Tilda. The share of live links by service ranged from 88% to 94%.

Server response codes

A page is gone when the server responds with 404 or 410. Codes 401, 403, 429 and 503 mean something else: the server did not serve the page to the bot, though the page itself may exist.

410 means the content was removed permanently and there is no redirect address. 403 means the client lacks access rights and the server refuses. 429 is rate limiting; the page may be intact. 503 is a temporary state; the server may specify recovery time in the Retry-After header. 401 can be a standard Cloudflare challenge response: the visitor has not been blocked.

Python check script

The script sorts addresses into the code table categories, plus two service ones: no response and parse manually. Save the code in check.py, addresses in urls.txt one per line, install the requests library and run in the terminal: python3 check.py urls.txt.

The script uses GET instead of HEAD, because HEAD returns only headers without the body and the script would not see the page text. GET leaves room to add content checks. The script follows redirects up to ten hops: Google crawlers by default follow chains up to ten hops. Timeout is five seconds for connection and fifteen for reading. Without it Requests may wait for a response indefinitely. The script identifies itself as a link checker and does not pose as a browser.

Why AI links to a deleted article

Most likely because the AI search index lags behind sites: the article was already removed but the crawl has not reached it yet. According to Pew Research Center, a quarter of pages that existed in 2013–2023 became unavailable by October 2023, and most often a single page on a working site was deleted. Perplexity writes that re-indexing every address frequently is impossible, and what and when to crawl is decided by a machine learning model. So a deleted page can remain in its index until the crawl reaches it. OpenAI publishes a timeframe only for robots.txt changes, around 24 hours, and does not state how long it takes to remove pages from ChatGPT search.

Google on 4xx responses except 429 reports that there is no content, and previously indexed addresses stop being used over time. Yandex removes a page with a 404, 403 or 410 response from the search index within a week after the bot discovers it, but time may pass before the crawl.

What to do before deleting a page

Before deletion, set up a redirect to a current page on the same topic. The instruction for whoever runs the site: configure a permanent redirect from the old address to the chosen new one and verify in a browser that the old link opens the new page. Then someone who clicks the old link from an AI response will land on live text. A redirect to the home page will also show as live to the script, but a reader from the AI response will not find what they were promised. If there is no replacement, serve 410: it is a clear signal that the content is gone for good.

In a Tow Center study on news citation, Gemini and Grok had fabricated or broken addresses in more than half of responses. In an arXiv preprint, a fabricated term on accessible platforms appeared in citations from 8 out of 10 AI search engines within a week.

TG X Copied
Mentioned: NeBlaskWilder

Project sponsors

Companies that keep this analytics open