Retrieval When the Query and Document Are in Different Languages

A user asks in Spanish; the manual exists only in German. There are two honest architectures for that, they fail in different places, and most teams pick one by accident rather than by deciding which failure they would rather have.

The problem, stated precisely

Retrieval works by putting query and document in a shared representation and measuring closeness. When they are in different languages, either the representation has to span both — which is a property of the embedding model — or something has to move one of them into the other’s language before the comparison. Those are the only two options, and every product feature in this area is one of them with details attached.

Note this is a different problem from the one in retrieval across a mixed-language corpus. There, the corpus contains the query’s language and the ranking is skewed. Here the corpus may contain no document in the query’s language at all, so filtering to the query language returns nothing and there is no ranking to fix.

Strategy one: translate, then retrieve

Put a translation step on one side of the comparison. Two variants, and they are not equivalent.

Translate the query

At query time, detect the language, translate the query into each language present in the corpus, run one search per language, merge. The index stays as it is.

• Cost is per query and grows with the number of corpus languages. Three languages means three translations and three searches on the critical path.
• Latency is the real constraint. A translation call is a full model round trip before retrieval can even start, so it is added to time-to-first-token, not overlapped with it. You can parallelise the searches; you cannot start them early.
• Queries are short, which is where translation is weakest. A three-word query has almost no context to disambiguate with, so an ambiguous term translates to the wrong sense and the retrieval is confidently wrong. This is the dominant accuracy failure of this variant.
• It is debuggable. You can log the translated query and see exactly what was searched for, which is worth more in production than a couple of points of accuracy.

Translate the documents

At ingest, translate every document into a pivot language — usually English — and index the translations alongside or instead of the originals.

• Cost is per document, paid once, and it is substantial for a large corpus. Re-paid on every re-ingest and on every document update.
• Zero added query latency, which is the strongest argument for it.
• Translation errors are baked in. A mistranslated passage is permanently wrong in the index, and the chunk that gets retrieved is the mistranslation rather than the source. Keep the original text in the payload and pass that to the generation model, so the translation is used for matching only.
• Index size and cost multiply by the number of languages you keep.

Strategy two: embed cross-lingually

Use a multilingual embedding model trained so that a sentence and its translation land near each other in one shared space. Index everything as-is, embed the query as-is, and compare directly. No translation step exists.

• No added latency, no added per-query cost. One embedding call, one search — the same pipeline you would run monolingually.
• Alignment quality is uneven across language pairs.A model trained with heavy Spanish–English parallel data aligns that pair well; a pair with little parallel data in training aligns weakly, and the model does not report which case you are in.
• The language-clustering baseline still applies. Same-language pairs score higher than translation pairs, so if any same-language documents exist they will outrank a better cross-language match. When the corpus is entirely in a language the query is not, this stops mattering — everything is cross-language, so the offset is uniform and the ranking within the corpus is unaffected.
• Named entities and technical terms carry it. Cross-lingual retrieval works best when the query contains error codes, product names or numbers that appear identically in both languages, and worst on purely conceptual queries.

The alignment property has its own page: how cross-lingual embedding alignment works. The practical point here is that alignment is a claim about the model, and it is a claim you should verify for your specific pair rather than accept from a model card’s aggregate score.

Choosing between them

• Corpus in one language, queries in many — cross-lingual embedding. Document translation would mean translating everything into every query language, which does not scale, and query translation adds latency to solve a problem the embedding model already solves.
• Corpus in many languages, queries in one — document translation into the query language, done at ingest. You pay once, query latency stays flat, and the top-k becomes comparable across the whole corpus.
• Low-resource language on either side — query translation. Where embedding alignment is weakest, a dedicated translation model is usually still workable, and you get a logged artefact to inspect when it is not.
• Strict latency budget — anything but query translation. It is the only one of the three that puts a model call in front of retrieval.
• Queries are long — query translation becomes much more attractive, because the accuracy problem was short queries, and a paragraph-length query translates well.

The hybrid that usually wins

Run the cross-lingual embedding search unconditionally, because it is free and already in the pipeline. Add query translation only as a second retrieval when the first one comes back weak — the top score below a threshold, or the top results clustered in a language other than the corpus you expected. Fuse the two candidate lists by rank rather than by score, since scores from two different retrieval paths are not on a comparable scale.

1. Embed the query as written; retrieve top 50.
2. If the best score clears your threshold, stop. Most queries stop here, so the added cost is paid on a minority of traffic.
3. Otherwise translate the query into each corpus language, retrieve top 50 for each.
4. Fuse all lists with reciprocal rank fusion, then re-rank the top 20 with a cross-encoder if the latency budget allows.
5. Pass the original-language chunk text to the generation model, never a translation of it, and instruct the model to answer in the query’s language.

That last step is where teams lose accuracy they had already won. Retrieval-side translation is a matching aid. The generation model is usually a better reader of the source language than your translation step is, so giving it the original text and letting it answer in the user’s language beats handing it a translation of the source.