LLMs in Search: Where They Help and Where They Don’t

“Should we use an LLM in our search?” has four different answers, because there are four different places to put one and they differ by three orders of magnitude in cost.

Four slots, not one

The mistake is treating these as one decision. The first and last are almost always worth doing; the second is a latency and cost question with a real answer; the third is a product question that has nothing to do with ranking quality.

Rewriting: the easy win

A user types four words with a typo and an implicit constraint. A model turns that into a well-formed query, or three of them, or an expanded version containing the vocabulary the documents actually use. This works because the bottleneck on tail queries is vocabulary mismatch, and generation is very good at producing plausible vocabulary.

The family of techniques is well documented and mostly predates the current model generation in spirit — Query2doc (Wang and colleagues, 2023) generates a pseudo-document to search with, and HyDE does the same thing for dense retrieval. Both are covered elsewhere: query rewriting and multi-query retrieval and HyDE. What this page adds is the operational reason it is the easy slot:

• The input and output are both tiny. A query is tens of tokens; a rewrite is tens of tokens. Cost per call is negligible compared with anything that reads documents.
• It caches perfectly. Queries follow a heavy-headed distribution, so a cache keyed on the normalised query serves the head almost entirely from memory. For the very top of the distribution you can precompute rewrites in batch and never make a live call at all.
• Failures are cheap. A bad rewrite produces bad results for one query. A bad reranking decision does the same, but a bad rewrite can be detected and reverted by falling back to the original query when the rewrite returns fewer results.

Ranking: what the literature actually shows

There is a real and active research line here, and it is worth reading rather than summarising into a verdict. The three strands to know:

• Encoder cross-attention rerankers. monoBERT (Nogueira and Cho) and monoT5 (Nogueira and colleagues) score a query-document pair with a single forward pass of a fine-tuned encoder. These are the models most commercial rerankers are, they run in milliseconds, and they are the baseline any LLM approach has to beat.
• Listwise permutation generation. Sun and colleagues (2023), “Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents”, prompts a general model with a numbered list of passages and asks it to output a permutation, using a sliding window to cover more candidates than fit in context. The result that made the paper notable is that a general instruction model, with no ranking-specific training, is competitive with supervised rerankers on standard benchmarks.
• Distillation. RankVicuna and RankZephyr (Pradeep and colleagues) distil that listwise behaviour into smaller open models, which is the obvious response to the cost problem the next section derives.

Two failure modes are documented in that line of work and both are structural rather than incidental. Listwise LLM rankers are sensitive to the order the candidates are presented in — the same set of passages in a different input order produces a different permutation — so a serious implementation shuffles and aggregates over multiple passes, which multiplies the cost below. And the model can emit a malformed permutation: a repeated identifier, a missing one, an identifier that was never in the input. That needs a repair step, and the repair step is where a naive implementation silently drops documents.

The arithmetic that decides it

Take the standard reranking job: 100 candidate passages, roughly 200 tokens each. Prices below are placeholders — put your own in, they move constantly.

ASSUMPTIONS
  passage length      200 tokens
  candidates          100
  window size         20 passages, stride 10  ->  9 windows
  price, input        $0.50 per 1M tokens
  price, output       $1.50 per 1M tokens
  cross-encoder       1,000 passages / second on one GPU

LISTWISE LLM RERANKER
  input per window    20 * 200 = 4,000 tokens (+ instructions)
  input per query     9 * 4,000 = 36,000 tokens
  output per query    9 * 60    =    540 tokens
  cost per query      36,000 * 0.5e-6  = $0.0180
                    +    540 * 1.5e-6  = $0.0008
                                       = $0.0188
  latency             9 sequential windows, ~1 s each = ~9 s

CROSS-ENCODER RERANKER
  latency             100 / 1,000 = 0.1 s
  cost                GPU time, not tokens

At a million queries a day the LLM reranker costs about $18,800 a day on those assumptions, and takes nine seconds. Neither number is survivable for interactive search, and the latency is the harder of the two because it cannot be fixed with money. Multiply by the two or three passes that the order-sensitivity above demands and it gets worse.

Compare the rewriting slot on the same assumptions: roughly 200 input and 40 output tokens, so about $0.00016 per call, cacheable, and one round trip. That is a hundred times cheaper than the reranker per query before the cache does anything, and it is why the two slots have different answers.

What the arithmetic actually implies is not “never rank with an LLM”. It is that an LLM ranker is affordable when the query volume is low and the value per query is high — internal knowledge search, legal and medical retrieval, an agent’s single research step — and unaffordable when volume is the point. The commercial cross-encoder rerankers covered in the reranking page exist precisely to occupy the gap between those two.

The offline slot nobody uses enough

The best value in the list is the slot with no latency requirement at all, because everything expensive becomes affordable when it is amortised over every future query.

• Generating relevance judgements. The binding constraint on relevance tuning is how many judged query-document pairs you can afford, and the sample size derivation there shows how badly you need hundreds. A model can produce a first pass that humans audit, which changes the economics of the whole loop. The judge-reliability caveats in LLM-as-a-judge apply unchanged, and position bias in a judge is the same phenomenon as position bias in a ranker.
• Document expansion. Generating the questions a document answers and indexing them alongside it attacks vocabulary mismatch from the document side instead of the query side. The cost is paid once per document. It is the same idea as contextual retrieval in a RAG pipeline.
• Synthetic training queries. Generate plausible queries for documents you have, and you have training pairs for a retriever or a reranker without a click log — which is the only way to train anything at all before launch.