Embedding Text vs Embedding Questions: The Asymmetry Problem

“What is the refund window?” and “Refunds are available within 30 days of purchase, provided the item is unused” are a perfect retrieval pair and are not similar texts. One is a seven-word interrogative; the other is a declarative clause three times as long with no overlapping content word. Asking one embedding function to place both in the same neighbourhood is asking for something the naive setup does not give you.

A question is not a short answer

Symmetric similarity — the thing sentence-similarity benchmarks measure — is about whether two texts mean the same thing. Retrieval asks something different: whether one text answers the other. These are separate relations, and a model optimised for the first is not automatically good at the second.

Models built for retrieval handle it by learning two slightly different mappings out of one set of weights, conditioned on a marker that tells the model which role this text is playing. Sometimes the marker is a literal prefix string prepended to the input; sometimes it is an API parameter that the provider turns into one. Either way the model has been trained with it present, and omitting it puts the model in a regime it never saw during training.

There is a second, blunter asymmetry underneath: length. Queries are short and documents are long, and the pooling that turns token vectors into one sentence vector behaves differently over 8 tokens than over 400. The role marker is partly the model’s handle on that too.

The conventions, by model family

The differences between rows are not cosmetic and the conventions do not transfer. Prepending E5’s “query: ” to a BGE model gives you a model processing a token sequence it has no training for; using BGE’s long instruction with E5 does the same. There is no universal prefix, and the only correct source is the model card for the exact checkpoint you are running — including its version, since these conventions have changed between releases within a family.

Cohere’s choice of a required parameter is worth dwelling on, because it converts a silent quality regression into a loud error. A prefix convention expressed as documentation gets applied at index time by the person who read the docs, and forgotten six months later in the new ingestion path written by someone who did not.

Detecting that you got it wrong

There is no error and no warning; retrieval simply gets worse. Two checks, both quick:

# 1. does the prefix change anything at all?
a = embed("query: what is the refund window")
b = embed("what is the refund window")
cosine(a, b)     # if this is 0.999, the model is ignoring the prefix
                 # and you are using a model that does not want one

# 2. does the right convention actually retrieve better?
for convention in [none, e5_style, bge_style]:
    index  = embed_all(docs, convention)
    print(convention, recall_at_10(gold_queries, index))

The second check is the one that settles it, and it needs the same gold set of fifty queries every other decision in this cluster needs. The asymmetry question is not a matter of taste — one configuration retrieves better on your data, and it takes twenty minutes to find out which.

One operational trap: the prefix must be applied consistently at index time and at query time, in the same version of the pipeline. A re-indexing job that applies the document prefix while the online query path forgets the query prefix produces an index that half-works — good enough not to page anyone, bad enough to lose real recall for months.

When you want symmetry instead

Not every use of embeddings is retrieval, and applying retrieval prefixes to a symmetric task is the mirror-image error.

• Deduplication. Both sides are documents and neither is a query. Use the document convention on both, or a clustering mode if the model offers one.
• Clustering and topic discovery. Nomic’s separate clustering prefix exists because the geometry that makes retrieval work — queries pulled toward their answers — is not the geometry that makes clusters clean.
• Classification against labelled examples. Comparing a new item to stored examples of the same kind is symmetric, and several families offer a classification mode for exactly this.
• Paraphrase and near-duplicate detection. Symmetric by definition, and often better served by a model trained on sentence similarity rather than on retrieval.

Closing the gap from the other side

Prefixes ask the model to bridge the query/document gap. Two published techniques instead move the texts closer together before embedding.

HyDE — Gao et al., Precise Zero-Shot Dense Retrieval without Relevance Labels (2022) — has a language model write a hypothetical answer to the query, then embeds that answer instead of the query. You are now comparing a document-shaped text to documents, which sidesteps the asymmetry entirely. It costs a generation call per query in both latency and money, and it inherits whatever the generator hallucinated, so it earns its place mainly where queries are unusual and the corpus is technical.

Document expansion works from the opposite end: generate, at index time, a handful of questions each chunk answers, and embed or index those alongside it. The cost is paid once during ingestion rather than on every query, which is the better trade for a stable corpus with heavy traffic.