Multilingual Embeddings: Cross-Language Retrieval

A multilingual embedding model puts “hoeveel kost verzending” and “shipping costs are calculated at checkout” near each other in the same space. That is a genuinely remarkable capability, and it comes with a specific bias that will make your evaluation look wrong until you know to control for it.

One space, many languages

There is no translation step. The model has one set of weights and one output space, and it was trained so that a sentence and its translation land close together — typically using parallel corpora as positive pairs in the same contrastive objective a monolingual model uses. What the model learns is that these two token sequences should produce the same vector, and it generalises from the pairs it saw to pairs it did not.

Two structural consequences follow, and both are consequential for system design. First, adding a language costs no infrastructure: one index, one model, one query path. Second, the model’s capacity is shared, so a multilingual model spends on a hundred languages the representational budget a monolingual one spends on one. That is the real reason multilingual models tend to trail specialist English models on English-only tasks — not a defect, an allocation.

Tokenisation compounds it. A vocabulary shared across scripts spends fewer tokens per word on high-resource languages than on low-resource ones, so the same sentence in a poorly-covered language costs more tokens, fragments into less meaningful subwords, and is represented worse. This is the mechanism behind most low-resource degradation, and you can observe it directly by tokenising the same paragraph in each of your languages and comparing counts.

Three architectures, three costs

The choice is usually made by one question: does a user searching in Dutch need to find a document that exists only in English? If yes, the single multilingual index is nearly forced, because language routing structurally cannot return it. If every document exists in every language, per-language indexes are the better-quality answer and the translation pipeline is unnecessary.

Language bias, the failure that surprises people

In a shared multilingual space, vectors cluster by language as well as by meaning. The consequence in retrieval is systematic: given a query in language X, documents in language X score higher than equally relevant documents in language Y, simply because they share the language component of the representation.

This is why a naive mixed-language index appears to work in testing and then disappoints. If your corpus is 80% English and a Dutch query arrives, the handful of Dutch documents can crowd out better English ones — or, with the proportions reversed, a relevant Dutch document never reaches the top 10 against a wall of English ones. Neither outcome is a relevance judgement; both are the language component asserting itself.

The mitigations are all structural rather than clever. Retrieve per language and fuse the ranked lists with reciprocal rank fusion, which removes the cross-language score comparison entirely. Or normalise scores within each language group before merging. Or, where the product allows it, filter to the user’s language and accept that cross-language recall is not a feature you are offering. The one thing not to do is compare raw similarity scores across languages and treat the ordering as relevance.

The other four failure modes

• Low-resource degradation. A model listing a hundred languages does not serve them equally. Coverage in the training data varies by orders of magnitude, and quality follows it. Check where your languages sit in whatever the model card discloses before assuming parity.
• Code-switching. Real user text mixes languages within a sentence — an English technical term inside a Dutch question is the normal case in engineering support. Language detection on such text is unreliable, which quietly breaks any architecture that routes on detected language.
• Script and transliteration. The same language written in two scripts, or transliterated into Latin, may not map to the same region of the space at all. If your users do this, test it explicitly; it will not show up in any public benchmark.
• Named entities and units. Product names, addresses and formatted numbers do not translate and are exactly where a lexical index outperforms any embedding. Hybrid retrieval matters more in multilingual systems, not less.

Building your own language-pair table

Public multilingual retrieval benchmarks exist and are worth reading — MIRACL (Zhang et al., 2022) covers 18 languages with human-annotated relevance judgements, and the multilingual portions of MTEB aggregate more. What none of them can tell you is how a model handles your language pairs on your content, and that is a table you can build in a day:

• Take 30 queries per language you support, drawn from real logs, with their known-correct documents marked.
• Embed the whole corpus once with the candidate model, then evaluate recall@10 for every combination of query language and document language. For five languages that is a 5×5 grid of numbers.
• Read the diagonal first — that is same-language retrieval and it should be your strongest. The off-diagonal cells are cross-lingual performance, and they are the ones that decide whether the single shared index is viable for you.
• Repeat with the translate-to-pivot architecture on the same queries. You now have a direct comparison of two designs on your data, which is a decision, rather than two published averages, which is not.

Keep the grid. When you next change models it is the artefact that tells you, in an hour, whether the change helped every market or helped the two largest at the expense of the rest — a distinction a single average score is specifically designed to hide.