Reranking: The Cheapest Accuracy Win in RAG

A reranker is a second model that looks at the query and one candidate document together and scores the pair. That single structural difference — together, rather than separately — is why it can rank better than the retriever that produced the candidates.

Why the second pass can be better

Your retriever is a bi-encoder. It embedded the document weeks ago, without knowing the query, and embedded the query without knowing the document. All the interaction between them happens in a single dot product at the very end. That is what makes it fast: the document vectors are precomputed, so a query is one matrix multiply over the whole corpus.

A cross-encoder concatenates query and document and runs them through a transformer together, so every token of the query can attend to every token of the document. It can notice that the query asks about version 3.11 and the passage is about 3.1. It can notice the negation. It outputs a single relevance score and nothing reusable, which is exactly why it cannot be the first stage: scoring a million documents means a million forward passes.

So the architecture is forced. Cheap recall-oriented retrieval to get from a million to fifty; expensive precision-oriented scoring to get from fifty to five. The reranker never has to see the corpus.

What the published results show

The result that established the pattern is Nogueira and Cho, Passage Re-ranking with BERT (arXiv:1901.04085, 2019). On the MS MARCO passage-ranking task they took BM25’s top-1000 candidates and rescored them with a BERT cross-encoder. The reported MRR@10 on the development set moved from roughly 0.17 for BM25 alone to roughly 0.35 with a BERT-base reranker and roughly 0.37 with BERT-large — the same candidate set, reordered, roughly doubling the metric.

Two things about that number are worth being careful with. It is a 2019 result on a specific academic benchmark with short passages and keyword-ish queries; your corpus is not MS MARCO. And MRR@10 rewards getting the single best passage to the top, which is precisely what reranking does and precisely what a retriever is worst at — so the benchmark flatters the technique. What generalises is the direction and the mechanism, not the magnitude.

The practical claim that follows is narrower and more defensible: if your evaluation shows that the right chunk is usually somewhere in the top 25 but often not in the top 5, a reranker is the correct fix, because that is exactly the error it addresses. If the right chunk is not in the top 50 at all, reranking cannot help you and the problem is upstream in chunking or in the query.

Late interaction, the middle option

There is a third architecture between the two. ColBERT (Khattab and Zaharia, SIGIR 2020) keeps a vector per token rather than per document, and scores a pair by summing, over query tokens, the maximum similarity against any document token. The document side stays precomputable, so it can serve as a first-stage retriever, but the matching is finer-grained than a single dot product.

The cost is the index. One vector per token instead of one per chunk is an order-of-magnitude storage increase before compression, which is why late interaction shows up in systems where retrieval quality is the product and rarely in systems where it is a feature.

The latency and cost budget

Reranking 25 candidates of 500 tokens each means pushing about 12,500 tokens plus the query through the reranker, in 25 independent forward passes that batch well. Hosted rerank endpoints typically bill per search rather than per token — a common shape is a fixed price per query covering up to some number of documents — which makes the cost per query flat in the candidate count up to that ceiling and a step function above it.

Assume, as an illustration you should replace with your vendor’s actual rate, $2.00 per thousand searches. That is $0.002 per query. Now compare it against the generation call it is protecting: 3,000 input tokens at $0.30 per million is $0.0009. At those assumed rates the reranker costs more than twice the generation step, which surprises people. It is still usually worth it — but it means reranking is a line item to model, not a free win, and it is the first thing to reconsider if your per-query cost is a problem.

Latency is the other budget. A hosted reranker adds a network round trip strictly between retrieval and generation; it cannot be parallelised with either because it depends on one and feeds the other. Budget 50–200 ms and decide whether your product can spend it. For a batch pipeline, obviously yes. For an autocomplete, no.

Choosing the candidate count

The one parameter that matters is how many candidates you rerank. Reason about it as two separate quantities:

• Recall@n of the retriever, where n is the candidate count. This is the ceiling — the reranker can only reorder what it was given, so any document missing at this stage is permanently lost.
• Precision@k of the reranker, where k is what you pass to the model. This is what you are buying.

Measure recall@n on your evaluation set at n = 10, 25, 50, 100. It rises steeply and then flattens; rerank at the elbow. Going past it costs money and adds latency to raise a ceiling that was no longer the binding constraint. In most corpora the elbow is somewhere between 25 and 50, but that is a property of your chunking, not a universal constant.

Hosted or self-hosted is the other decision. Cross-encoder rerankers are small by language-model standards — the published research models are in the hundreds of millions of parameters, not the billions — so serving one yourself is genuinely feasible on a modest GPU, and open rerankers are available. The trade is the usual one: a hosted endpoint is a line item and somebody else’s uptime, a self-hosted one is a fixed cost that amortises and an on-call rotation. At the volume where the per-search fee starts to dominate your bill the arithmetic tips, and the break-even is worth computing rather than assuming.

One last property to watch. A reranker scores relevance, not sufficiency — it will happily rank a chunk that discusses the query terms above one that answers the question, if the first is more topically on-point. That is correct behaviour for a ranking model and an imperfect fit for RAG, where what you want is the chunk that resolves the question. It is a reason to keep more than one or two chunks after reranking even when the top score looks decisive, and a reason not to read the reranker’s score as a confidence that the answer is present.