How Modern Search Works: Retrieve, Rank, Rerank

Every search system large enough to have a problem ends up with the same shape: a cheap stage that reads the whole corpus, a middling stage that reads a thousand documents, and an expensive stage that reads fifty. That is not a convention. It falls out of multiplying two numbers together.

The three stages

Retrieval — sometimes called candidate generation — takes the query and produces a few hundred to a few thousand documents out of a corpus that may hold millions. Its job is recall: get the good documents into the set, in any order. An inverted index with BM25 does this, an approximate nearest-neighbour index over embeddings does this, and most serious systems run both and merge.

Ranking takes those candidates and orders them using features that were too expensive to compute over the whole corpus: freshness, popularity, click history, query-document match features, business rules. Classically this is a learning-to-rank model — a gradient-boosted tree ensemble like LambdaMART is still the workhorse because it trains on a laptop and scores a thousand documents in single-digit milliseconds.

Reranking takes the top tens of documents and scores each one jointly with the query using a model that reads both together. A cross-encoder is the standard choice and it is genuinely much better than anything upstream, because it can attend to the query while reading the document rather than comparing two independently produced vectors. It is also one to three orders of magnitude more expensive per document, which is why it never sees more than a few dozen. The mechanics are in the reranking page; what matters here is where it sits and why.

The boundaries between the three are conventions rather than laws, and real systems blur them constantly. Stage one is frequently two retrievers run in parallel and fused, which hybrid search covers. Some systems collapse stages two and three into a single model. Others add a fourth stage after the reranker for business rules and diversification, because those decisions need to see the final list. What survives every variation is the shape — expensive scoring applied to progressively fewer documents — and the two properties below, which hold no matter how many boxes the diagram ends up with.

Why the funnel is not a choice

Write the per-query cost of a cascade with three stages, where stage 1 scores the whole corpus, stage 2 scores k1 documents and stage 3 scores k2:

cost = c1 * N  +  c2 * k1  +  c3 * k2

N   total documents in the corpus
k1  candidates handed to the ranker
k2  documents handed to the reranker
cN  cost to score one document at that stage

Now put in some numbers, all of them assumptions you should replace. Take a corpus of N = 10,000,000 documents and suppose one GPU runs a cross-encoder over roughly 1,000 short passages per second. Scoring the whole corpus with the cross-encoder takes 10,000,000 / 1000 = 10,000 seconds, or a little under three hours — per query. Scoring the top 100 takes 0.1 seconds. The ratio is 100,000 to one, and it is the entire justification for the architecture.

The formal version of this argument is the cascade ranking model of Wang, Lin and Metzler (2011), which treats the choice of stages and cut-offs as a joint optimisation of effectiveness against a cost budget. The practical version is that the first stage exists to make the third stage affordable, and it earns its place by being wrong cheaply rather than by being right.

Recall is a ceiling, not a target

Here is the property people learn the expensive way. A later stage can only reorder what an earlier stage handed it. So the recall of the whole system is bounded above by the recall of stage one, and no amount of reranking can recover a document that was never a candidate.

Work it. Suppose a query has 5 genuinely relevant documents in the corpus, and stage one’s top-1,000 contains 4 of them. Then recall@1000 for stage one is 0.80. The best possible recall@10 of the finished system is also 0.80 — the reranker would have to place all four in the top ten and the fifth simply does not exist as far as it is concerned. If your end-to-end recall@10 is 0.55, the gap between 0.55 and 0.80 is a ranking problem and the gap between 0.80 and 1.00 is a retrieval problem, and those have completely different fixes.

This is why the single most useful diagnostic in a search system is recall of the first stage at its own cut-off, measured separately. Teams that only track end-to-end nDCG spend months tuning a reranker against a ceiling they cannot see. Evaluating retrieval and generation separately makes exactly the same argument for RAG pipelines, which are cascades with a language model bolted on the end.

Choosing k at each stage

There is a real procedure for this and it does not involve guessing. Fix your end-to-end quality metric. Then, holding everything else constant, sweep k1 and plot stage-one recall@k1 against it. That curve is almost always steeply concave: it climbs fast, then flattens. The right k1 is where the curve flattens, because every candidate past that point costs the ranker time and buys nothing.

Do the same for k2, but against end-to-end nDCG rather than recall, because the reranker’s job is ordering. And notice the asymmetry: raising k1 costs cheap ranker time, while raising k2 costs expensive reranker time. If the latency budget is tight you widen the top of the funnel and narrow the bottom, never the reverse.

Where cascades break

• The filter applied after retrieval. Retrieving 100 candidates and then filtering by permission, stock or language leaves you with whatever survives, which for a selective filter is often nothing. The arithmetic is worked out in the page on permission filtering and the fix is always to push the filter into stage one.
• Stages optimised against different labels. If the retriever was trained on one relevance definition and the reranker on another, the reranker spends its capacity undoing the retriever. Both stages should be trained against the same judgements.
• The reranker that only sees agreement. When stage one is good, the top 50 are often near-identical in quality and the reranker has nothing to do. That is not a reason to remove it — it is a reason to feed it a wider, more diverse candidate set, which is where diversification at the candidate stage earns its keep.
• Latency measured as a mean. A cascade’s tail is dominated by whichever stage fans out, and a mean hides it entirely. Track p95 per stage, not overall.