How AI Assistants Choose Which Pages to Cite

An assistant that shows five links did not rank the web. It ran a short pipeline of ordinary components — a search backend, a fetcher, a chunker, a token budget and a generation step — and about half of that pipeline leaves evidence on your server. This traces the whole of it and marks each stage as observable or not.

Two paths, and only one leaves evidence

When an assistant answers a question about your company, one of two things happened, and they are not variants of each other.

On the parametric path the model answered from its weights. Nothing was fetched, nothing was searched, and your server saw no request. Anything it said about you came from text in the training corpus, which may be years old, may be a third-party summary of you, and may be about a different company with a similar name. If it produced a URL on this path, the URL was generated token by token like any other string, which is why plausible links to pages that never existed are a recognised failure mode rather than a bug.

On the grounded path a retrieval step ran first. Something issued search queries, fetched documents, put their text into the context window and asked the model to answer using it. That path touches your server, and everything in this page that you can verify is verifiable because of that touch.

The first diagnostic question about any assistant claim is therefore which path produced it, and you can usually tell: a grounded answer shows sources and appears after a visible delay, and a parametric one appears immediately with no links. This matters because the two failure modes have completely different remedies, which is the subject of correcting something an assistant says about you.

The grounded pipeline, stage by stage

Implementations differ, but the stages are the same ones any retrieval-augmented system has, because that is what these are. In order:

Every stage is a filter. A page that survives to the citation list survived all seven, and failing any one of them is invisible from outside — there is no error message and no report. Most of the useful work in this field is working out which stage you are failing.

The search backend is upstream of everything

The retrieval stage is a web search. Assistants source it in one of three ways: they license a commercial search API, they operate their own crawler and index, or they do both and merge. The consequence is the same in each case and it is the most under-stated fact in this field: if the backend index does not return your URL for the rewritten query, nothing downstream can cite you. No amount of page-level optimisation reaches a document that was never in the candidate list.

That makes conventional indexing hygiene the floor rather than the legacy concern it is often presented as. Being crawlable, being in a sitemap, resolving one canonical URL per topic and ranking for the long-tail phrasing a rewritten question produces are all prerequisites — see retrieval-friendly site architecture. The queries a rewriter produces tend to look like long-tail queries: longer than a human types, explicitly scoped, often with a qualifier the user only implied.

The context budget decides how many sources survive

The assembly stage has a hard limit, and you can reason about it arithmetically even without knowing the exact number. Suppose the system allocates 8,000 tokens of context to retrieved material. At roughly four characters per token for English prose, that is about 32,000 characters. If the chunker emits 600-token chunks, the budget holds about thirteen chunks in total — across every document it fetched, not per document.

budget            8,000 tokens of retrieved context
chunk size          600 tokens
chunks that fit    8,000 / 600  ≈ 13
documents fetched  10
chunks per doc     13 / 10  ≈ 1.3

So the realistic unit of citation is one passage, not one page. Your page does not compete as a page; a two-hundred-word span of it competes against a two-hundred-word span of nine other pages. That single fact explains most of what is useful in writing so a passage stands alone, and it explains why long, well-organised pages with one answer per section outperform the same words spread over five thin URLs.

The numbers above are an illustration of the shape, not a measurement of any product. No operator publishes its budget or chunk size, and they differ between the fast and slow modes of the same assistant. What does not differ is that a budget exists and that it is small relative to the web.

Citations are attached, not derived

This is the stage most explanations get wrong. The model does not record which document caused which word. It generates text, and citation markers are part of that generated text — produced by the same next-token process as everything else. A marker is therefore a prediction about which source is relevant, not a record of provenance.

Two consequences follow, and both are observable in the wild. A sentence can carry a citation to a source that does not contain the claim, because the source was in context and looked topical. And a sentence that is supported by a fetched document can go uncited, because the marker was simply not emitted. Systems mitigate this with a post-hoc verification pass — re-checking each sentence against its cited chunk with a second model call — which reduces the first failure and does nothing for the second.

If you are building this rather than being subject to it, the same problem in your own stack is covered in making retrieval-augmented answers cite correctly and grounding a model in sources.

What you can observe from your own server

Four things, all tier-2 observable, none requiring anything from a vendor:

1. Whether a fetch happened at all. Ask an assistant a question your site is the obvious answer to, then look at your access log for the following sixty seconds. A user-initiated fetch arrives with a distinct agent token — ChatGPT-User, Claude-User, Perplexity-User and their equivalents are separate from the same operators’ bulk crawlers precisely so that this is distinguishable.
2. Which URL it chose. The path in that log line tells you which of your pages the backend surfaced, which is a direct readout of the retrieval stage.
3. What it received. The status code, the byte count and the content type. A 200 of 900 bytes to a fetcher means it got a shell, not the article.
4. Whether a visitor followed. Referrers from assistant hostnames appear in the same log, which is the only click data in this field that is genuinely yours.

The mechanics of separating those lines from ordinary traffic, and of confirming an agent is who it says it is, are in finding out which AI bots visit you.

What nobody outside the lab can observe

Stated plainly, because everything sold in this field is sold as though these were known:

• The ranking function of the backend. Whether your page was candidate three or candidate thirty is not exposed.
• The chunker. Chunk size, overlap, and whether it respects headings.
• The selection score. Why one of your passages was kept and another dropped.
• The budget. How many chunks made it, and from how many documents.
• Causality. Whether a citation next to a sentence is the source the sentence came from.

Anyone claiming to have optimised for one of these is claiming access to something no public documentation describes. The honest posture is to optimise the stages you can see and to describe the rest as what it is.

What follows for a site owner

Three things, in order of how much they matter. First, be in the backend index at all, because retrieval is upstream of every tactic and nothing downstream compensates for absence. Second, serve the fetcher the actual text in the first HTTP response, because the fetch stage is where most sites silently fail — what a crawler sees when the page needs JavaScript covers how to test that in one command. Third, write passages that survive being extracted alone, because a passage is the unit that gets selected.

Everything beyond those three is, at the time of writing, a claim about the unobservable stages. That does not make it false. It makes it something you should not pay for as though it were established.