Why Embeddings Cluster by Language Before They Cluster by Meaning

Project a multilingual embedding space to two dimensions and you get neat islands, one per language, with topic structure visible only inside each island. This is not a defect that slipped through. It is what the training objective asked for, and knowing that tells you which of the available fixes can work.

Nothing in the objective forbids it

A multilingual encoder pretrained with masked language modelling is shown text in many languages and asked to predict missing tokens. The corpora are usually concatenated monolingual data, not translations. At no point does the loss compare a sentence with its translation, so at no point is the model penalised for representing them differently.

Cross-lingual ability in such a model is emergent, not trained: it comes from shared subword units, shared named entities and incidental parallel text in the crawl. That is enough to produce partial alignment, and it is nowhere near enough to overcome language identity, because nothing is pushing against it.

Models trained specifically for cross-lingual retrieval do push against it, with parallel data and a contrastive or distillation objective — that is the subject of how cross-lingual alignment works. Even those inherit the clustering to a degree, because the parallel data covers a fraction of the language pairs and domains the model is then used on. The force is applied where the data was and absent everywhere else.

Language is the highest-variance feature

Consider what a representation has to encode. Topic, register, sentiment, syntax, entities — and language. Of those, language is the easiest to predict from surface form and the most consistent across an entire document. Every token of a French document votes for French. In a learned space, a feature that is strong, consistent and cheap to compute ends up occupying a large share of the variance, and the directions with the largest variance are the ones cosine similarity is most sensitive to.

The published characterisation is Libovický, Rosa and Fraser, “How Language-Neutral is Multilingual BERT?” (2019). They analyse the representations as decomposing into a language-specific component and a language-neutral one, and show that the language-specific part is strong enough to identify a sentence’s language accurately from its embedding — and, crucially, that removing an estimate of it improves cross-lingual sentence retrieval substantially. That decomposition is the reason the fix below works at all: the language signal is largely a shift, not a tangle.

There is a second, compounding effect. Conneau and colleagues named the curse of multilinguality in “Unsupervised Cross-lingual Representation Learning at Scale” (2019): at fixed model capacity, adding languages improves the low-resource ones up to a point and then degrades every language, because capacity is finite and shared. A model spending capacity on keeping languages distinguishable has less left for distinguishing meanings within each.

What it does to top-k

The consequence is specific and it is worth stating in terms of the number you actually set. Suppose your corpus covers six languages in roughly equal volume and you retrieve the top ten chunks for a French query.

Because same-language similarity carries a constant bonus, French chunks fill the list first. The French chunks that are only loosely relevant still outrank the German chunk that answers the question exactly. With ten slots and thousands of French candidates, the ranked list can be entirely French, and the German answer might sit at rank forty. Your retrieval did not fail to find it; your cutoff removed it.

The symptom this produces at the application layer is a model that confidently says the information is not in the corpus, in a system where the information is in the corpus, in another language. There is no error and no low-confidence signal — the retrieved chunks are genuinely on-topic, just not the ones that contain the answer. Raising k dilutes the context rather than solving it, because the same bonus applies at every rank.

Subtracting the language mean

If language identity is largely an offset — a direction each language’s vectors are collectively shifted along — then estimating that offset and removing it should leave the semantic structure intact. That is what the 2019 analysis found, and it is cheap to implement.

1. Sample a few thousand chunks per language from your own corpus. A few hundred is enough for a stable mean; more is better and costs one embedding pass.
2. Compute the mean vector per language and store it. This is your language centroid table, a handful of vectors.
3. At index time, subtract the centroid of the chunk’s language from its vector, then renormalize to unit length.
4. At query time, do the same with the query’s detected language. Both sides must be centered or the geometry is inconsistent and results get worse, not better.
5. Evaluate on a cross-lingual probe set before and after, per language pair. Centering is not free and you need to see the trade.

The costs are real and you should know them before shipping this. Centering removes some genuine signal along with the language offset, so same-language retrieval typically gets slightly worse while cross-language retrieval gets substantially better. It requires a language label for every chunk and every query, so it inherits every weakness of language detection — and short queries and code-switched text are exactly where detection is least reliable. A wrongly labelled query gets the wrong centroid subtracted, which moves it somewhere arbitrary. Gate on detector confidence and skip centering below a threshold; see choosing a language detection confidence threshold.

When centering is not the answer

• Retrieve per language, then merge. Partition the index by language, take the top n from each, and fuse by rank. This guarantees every language is represented in the candidate set without touching the vectors at all, and it needs no centroids. It is the right default when you know which languages matter and the count is small.
• Translate the query. Issue the query in each corpus language and union the results. This sidesteps the geometry entirely and is often the highest-quality option, at the cost of one small model call per language per query.
• Rerank with a cross-encoder. A cross-encoder reads the query and the candidate together rather than comparing two independent vectors, so it does not have a language-offset problem. Over-retrieve widely, rerank the top hundred. This is the most reliable fix and the most expensive.
• Do nothing, deliberately. If users only ever want results in the language they asked in — which is true for most consumer search — the clustering is a feature. Filter by language metadata and stop worrying about it. Fixing this when you did not need it fixed costs you same-language quality for nothing.