Reranking Demo: What Rerank Depth Can and Cannot Fix

Runs BM25 over your own documents, then shows the ceiling reranking is bounded by — because a reranker can only reorder what the first stage handed it.

QueryScored against the documents below with BM25 (k1 = 1.2, b = 0.75), run in this page.

Documents — one per line, prefix a line with * if it is genuinely relevantYour relevance marks are the ground truth. The example set below is illustrative; replace it with your own corpus.

* Rotating an API key with no downtime: create the new key, deploy it everywhere, confirm traffic has moved, and only then revoke the old one.
Invoices are issued monthly and can be downloaded as PDF from the billing page at any time.
* An overlap window during which both the old key and the new key are accepted is what makes a rotation invisible to callers.
API keys are long-lived by default. Every key you have created is listed on the keys page with its last-used timestamp.
Rate limits are applied per key, so a newly created key starts with a fresh limit window.
SSO setup: map your identity provider groups to roles before you invite anyone, or the first login lands in the wrong role.
* If a key has leaked, revoke it immediately and accept the downtime. Graceful rotation is for planned changes, not for a compromise.
Webhook signing secrets are separate from API keys and are rotated on their own schedule.
Changing your account password does not invalidate any API key.
The keys page shows only the last four characters of a key. The full value is shown once, at creation.

Rerank depth k — how many first-stage results you send to the rerankerThe one control. Everything below position k is invisible to the reranker, permanently.

top 4

Final results kept (n)What you actually put in the prompt.

Rerank price per 1,000 documents scoredIllustrative default — type your reranker's own price.

Best possible recall@3 after reranking at depth 4

100%

Every relevant document is inside depth 4, so a perfect reranker could reach 100%. A real cross-encoder will land somewhere below this ceiling.

Documents in the corpus: 10
Marked relevant by you: 3
Recall@3 from BM25 alone: 100%
Relevant documents inside depth 4: 3 of 3
Relevant documents already lost: 0
Recall@3 ceiling after reranking: 100%
Documents reranked per query: 4
Rerank cost per 1,000 queries: $0.20

First-stage ranking — the reranker's horizon is the dashed line

13.88Rotating an API key with no downtime: create the new key, deploy it everywhere, confirm traffic has moved, and only then revoke the old one.rel
21.89An overlap window during which both the old key and the new key are accepted is what makes a rotation invisible to callers.rel
31.82If a key has leaked, revoke it immediately and accept the downtime. Graceful rotation is for planned changes, not for a compromise.rel
41.58Changing your account password does not invalidate any API key.
depth 4 — nothing below here is ever seen
51.17API keys are long-lived by default. Every key you have created is listed on the keys page with its last-used timestamp.
60.97Webhook signing secrets are separate from API keys and are rotated on their own schedule.
70.56Rate limits are applied per key, so a newly created key starts with a fresh limit window.
80.38The keys page shows only the last four characters of a key. The full value is shown once, at creation.
90.00Invoices are issued monthly and can be downloaded as PDF from the billing page at any time.
100.00SSO setup: map your identity provider groups to roles before you invite anyone, or the first login lands in the wrong role.

What this assumes: the first stage is BM25 over the documents you pasted, treated as the whole corpus — a real index has millions of documents and different IDF values, so your absolute scores will differ, though the ordering logic is identical. The “after” figure is a ceiling, not a prediction: it assumes a reranker that sorts your marked-relevant documents to the top perfectly. Real cross-encoders do not, so read it as the most that reranking at this depth could possibly buy you. Cost scales linearly with depth because rerankers score every candidate independently.

What the ceiling is for

Reranking is usually described as an accuracy win, which it is, and almost never described in terms of what bounds it, which is the part that decides whether it will help you. A reranker is a second pass over a candidate list. It cannot retrieve. Every document your first stage failed to return is gone before the reranker exists, and the best possible outcome of any reranking setup is therefore recall@k of the first stage — where k is the depth you send, not the number you keep.

That is why the slider is the whole tool. Teams add a reranker, see a modest improvement, and conclude reranking is overrated. Often what happened is that they sent it the top 10, and half the relevant documents were sitting at positions 15 to 40. Drag the depth up and the ceiling moves; drag it down and watch it collapse, usually faster than you expected.

The cost row is the other half of the decision. Reranking cost is linear in depth and independent of how many documents you keep, so depth 100 costs ten times depth 10 for the same three chunks in the prompt. Somewhere between those is the point where the ceiling stops rising and you are paying for reordering documents that were never going to matter. The honest way to find it is to mark relevance on 30 or 40 of your own queries, sweep the depth, and read where the ceiling curve flattens.

What this does not tell you is how close to the ceiling your reranker actually gets. That needs the reranker, your data and a labelled set — this page gives you the bound the measurement will sit under.

Reranking Demo: What Rerank Depth Can and Cannot Fix

What the ceiling is for

The reasoning behind this

Related tools