Query Rewriting and Multi-Query Retrieval

Retrieval quality is usually attacked from the index side. The other side is free and largely ignored: the string you search with does not have to be the string the user typed.

The gap between a question and a query

Embed the following and see what you get back: “what about the second one?”. There is nothing in it. It has no content words, no entity, no topic — its meaning lives entirely in the preceding turn, and the retriever has no preceding turn.

This is the normal case in a chat interface, not an edge case. Real user input in a conversational product is elliptical, pronominal, misspelt, mixes a question with a complaint, and frequently asks two things at once. A retrieval system that embeds it verbatim is searching with a degraded signal for no reason, when a small model can repair the string for a fraction of the cost of the answer.

The technique has a research lineage — Ma et al., Query Rewriting for Retrieval-Augmented Large Language Models (arXiv:2305.14283), framed it as a trainable rewriter sitting between the user and a frozen retriever — but the version worth shipping first is a prompt.

Four rewrites that do different jobs

Decomposition is the one that changes results most dramatically and the one most often skipped, because a compound question does not look broken. It retrieves something plausible, the model writes a confident half-answer, and nobody notices that the second clause was never served.

The rewrite step, in full

REWRITE = """You rewrite user questions into search queries for a
document index. You never answer the question.

Rules:
- Resolve every pronoun and ellipsis using the conversation.
- If the question asks more than one thing, emit one query per thing.
- Emit 1 to 3 queries. Fewer is better when the question is already
  a good query.
- Keep exact identifiers, error codes, version numbers and product
  names verbatim. Do not paraphrase them.

Return JSON: {"queries": ["...", "..."]}

Conversation so far:
{history}

Latest user message:
{message}
"""

def rewrite(history, message):
    out = small_model(REWRITE.format(history=history, message=message))
    qs  = json.loads(out)["queries"][:3]
    # Always keep the original. The rewriter is another thing that
    # can be wrong, and this bounds the damage when it is.
    return list(dict.fromkeys(qs + [message]))

Two details in that prompt earn their place. “You never answer the question” is there because a model handed a question will answer it unless told not to, and you will find answers in your query log. The instruction to keep identifiers verbatim is there because rewriters helpfully normalise ERR_TLS_CERT_ALTNAME_INVALID into “TLS certificate name error”, destroying the one token that would have matched exactly.

Keeping the original query in the list is the cheapest insurance in the pipeline. If the rewriter misreads the question, the unmodified query is still in the fan-out and its results still enter the merge.

Merging without double-counting

Now you have three or four ranked lists and need one. Do not concatenate and dedupe by first appearance — that ranks by which query happened to run first. Use reciprocal rank fusion, the same mechanism hybrid search uses to combine BM25 with vectors, since the problem is identical: several rankings, no comparable scores.

queries  = rewrite(history, message)          # 1-4 strings
rankings = [search(q, n=25) for q in queries] # run these concurrently
context  = rrf(rankings, k=60, n=8)

A chunk that surfaces for two different phrasings of the question accumulates score from both lists and rises, which is the behaviour you want. A chunk that surfaces only for the one variant the rewriter hallucinated stays low.

One caveat for decomposition specifically: fusing a decomposed question’s sub-queries can crowd out the smaller sub-topic. If “does it include SSO” returns two chunks and “what does it cost” returns twenty, fusion will hand the model a context that is mostly pricing. When you decompose deliberately, reserve slots per sub-query rather than fusing into one pool.

What it costs and when to skip it

One small-model call of a few hundred tokens, plus n embedding calls instead of one, plus n times the retrieval work. Against a generation step of a few thousand tokens, the rewrite is typically a small fraction of the request’s cost. The real budget is latency: the rewrite is strictly sequential before retrieval, so it adds a full round trip on the critical path.

• Always rewrite in a multi-turn interface. Without decontextualisation, turn two onwards is searching with noise.
• Skip the fan-out when the first turn is already a well-formed keyword query. A user who typed a function name does not need three paraphrases of it; they need it searched exactly.
• Cache the rewrite keyed on the conversation tail plus the message. Retries and edits hit it constantly.
• Log both strings. Query logs that only record the rewritten form make retrieval failures impossible to debug, because you cannot see whether the user or the rewriter caused them.

Evaluate the rewriter as its own component, because it is one. The measurement is straightforward: take your labelled question set, run retrieval with the raw query and with the rewritten set, and compare recall@k. If recall does not move, you have bought a round trip and a new failure mode for nothing — which happens more often than the technique’s popularity suggests, particularly on single-turn interfaces where the user’s question was already a serviceable query.

Where it reliably does move is the second and later turns of a conversation, and that is exactly the case a normal evaluation set never contains, because evaluation sets are written as standalone questions. If you build only one thing from this page, build a handful of multi-turn cases where the follow-up is unanswerable in isolation. They expose a gap nothing else in your test suite can see, and they are the cases your users hit constantly.