HyDE: Retrieving With a Hypothetical Answer

HyDE asks a model to invent an answer to the user’s question, throws the answer away, and searches with its embedding. It sounds like a joke about hallucination. It is a fix for a real geometric problem.

The asymmetry HyDE exploits

A question and its answer are not similar texts. “How long do refunds take?” is nine words, interrogative, with a subject the document never states in those terms. The passage that answers it is a paragraph of declarative prose beginning “Refunds are issued to the original payment method within 14 business days of approval”. They share one content word.

An embedding model trained on generic text similarity places texts near each other when they read alike. Questions cluster with questions. Documents cluster with documents. Searching a corpus of documents with a question vector means searching from a region of the space that the corpus does not occupy — and then hoping the ranking still comes out right.

Retrieval models trained specifically for this — DPR (Karpukhin et al., arXiv:2004.04906) and the asymmetric embedding models that followed — solve it by training two towers with question-passage pairs so the geometry is aligned by construction. That is the right fix when you have such a model and it covers your domain. HyDE is the fix when you do not.

The method

Gao et al., Precise Zero-Shot Dense Retrieval without Relevance Labels (arXiv:2212.10496), proposed it in 2022. Ask a model to write the document that would answer the question; embed that; search with it. The generated text will contain factual errors, and that is fine, because it is never shown to anyone — its job is to land in the right neighbourhood of the vector space.

HYDE = """Write a short passage from an internal knowledge base that
would answer the question below. Write it as documentation, not as a
reply: declarative, specific, no hedging, no "I". About 80 words.
It does not need to be true.

Question: {q}
"""

def hyde_search(q, n=25, samples=3):
    # Several samples at nonzero temperature, because one hypothetical
    # is one point and three cover more of the plausible region.
    docs = [small_model(HYDE.format(q=q), temperature=0.7)
            for _ in range(samples)]
    vecs = embed(docs + [q])              # keep the real question too
    v    = normalise(vecs.mean(axis=0))   # centroid of hypothetical + real
    return search_by_vector(v, n=n)

Two implementation choices there are load-bearing. Averaging several samples rather than trusting one reduces the influence of any single bad generation — the paper itself samples multiple hypothetical documents and averages. And including the real question’s vector in the average anchors the centroid to something that is definitely on topic, which is the cheapest available guard against the failure in the next-but-one section.

What the paper actually claims

The claim is narrower than the way HyDE is often repeated. The paper evaluates zero-shot dense retrieval — the setting where you have no labelled query-document pairs for your domain — and reports that HyDE on top of an unsupervised retriever (Contriever) beats that retriever substantially, and is competitive with fine-tuned models on several tasks and languages.

It does not claim to beat a well-fine-tuned in-domain retriever. If you have a modern asymmetric embedding model that was trained on question-passage pairs covering your domain, the asymmetry HyDE corrects is already corrected, and adding a generation step to the front of your retrieval buys latency and little else. This is the single most common reason a team tries HyDE and sees nothing.

How it breaks

The failure mode is specific and worth stating precisely, because it is not “the hypothetical document was wrong”. Wrongness in general is tolerated by the method. What is not tolerated is a confident wrong entity.

Ask “how do I rotate an API key?” about a product the model has never heard of, and it will write a fluent passage about a method called keys.rotate in a dashboard section called Security. Those invented names are rare, high-information tokens, and they dominate the embedding. The search now goes looking for a thing that does not exist, and it does so with more conviction than the original question ever had. The original question, for all its vagueness, was at least about your product.

• Proprietary vocabulary — internal tools, product names, private APIs. The model has no grounding and invents.
• Post-cutoff facts — anything that changed after the generating model’s training data ends. The hypothetical will describe the old world confidently.
• Queries that are already exact. Handed a specific error code, HyDE wraps it in a paragraph of generic prose and dilutes the one token that mattered. Detect identifier-shaped queries and route them straight to lexical search.

When to reach for it

Order of operations, given a fixed budget of engineering attention: hybrid search first, because it is cheap and its failure modes are understood; then reranking, because it addresses the largest error class; then, if retrieval is still missing documents whose vocabulary differs from the question’s, HyDE.

It sits naturally as one more arm in a multi-query fan-out: run the literal question, a rewritten question and a hypothetical document concurrently, fuse with RRF. That framing removes most of the risk, since a poisoned hypothetical contributes one ranking out of three rather than being the entire query. It also costs one extra generation call and no additional latency if the arms run in parallel — the hypothetical generation is the slow one, so the rest are free.

Budget it honestly, though. HyDE puts a generation call on the critical path before retrieval, and a hypothetical document of eighty words is eighty sequential decoding steps. Three samples at 80 tokens each, generated in parallel, still cost you the latency of one such generation before the search can even begin — and then the search, then the answer. In a streaming chat interface that is a visible delay before the first token, spent on a document nobody will read.

A cheaper variant worth trying first: instead of a full hypothetical passage, ask the model for the terms a good answer would contain — five to ten domain words and likely section headings. It is a much shorter generation, it lands well in the lexical arm of a hybrid retriever where the exact terms match directly, and it carries less risk of a fabricated entity dominating an embedding, because there is no surrounding prose for a wrong name to anchor.