Retrieval Across a Corpus That Mixes Several Languages

You index 40,000 English support articles and 3,000 Japanese ones in one collection with one multilingual embedding model. A Japanese user asks a question whose exact answer exists in Japanese, and the top ten results are English articles about roughly the right topic. Nothing is misconfigured. This is what a shared vector space does when the corpus is unbalanced.

The symptom

The signature is specific enough to recognise. Retrieval is not random — the returned chunks are topically plausible — but they are systematically in the corpus’s majority language, and the minority-language chunk that answers the question exactly sits at rank 40 with a similarity score just below the English cluster. Filtering the same query to the minority language alone returns the right chunk at rank one, which proves the chunk is fine and the ranking is the problem.

The generated answer then makes it worse rather than better. The model is handed English context for a Japanese question, and it will usually answer in Japanese from English source material, so the output looks correct and is sourced from a document that was the second-best answer available.

Why language dominates the ranking

Multilingual embedding models are trained so that translations land near each other, and they largely succeed at that — but “near each other” is relative. Within the same vector space, the similarity between two same-language texts on the same topic is typically higher than between a text and its own translation. Language identity is a strong, consistently present signal, and the model encodes it because it is genuinely part of what the text is.

The result is that the space is organised into language regions with topic structure inside each. A query vector lands inside the region of the language it is written in, and cosine similarity to anything in that region starts from a higher baseline than similarity to anything outside it. This is the mechanism described in why embeddings cluster by language, and it is not a defect to be tuned away — it is the same property that makes language identification from embeddings trivially easy.

Now add the count. Top-k is a global ranking over the whole collection. If 93% of chunks are English, then even a mild per-chunk advantage for English is amplified by there being thirteen English chunks competing for every Japanese one. The top ten fills with English before a Japanese chunk gets a look in, regardless of the individual scores.

What compounds it

• Hybrid search with a lexical component. BM25 over a mixed index computes IDF across all languages at once. A term that is common in Japanese but absent from the English 93% gets a very high IDF, which sounds helpful and mostly produces erratic scores that do not combine sensibly with the dense side.
• Chunk size differences. If the chunker was tuned in characters, the Japanese chunks are larger in tokens and therefore more diluted, which pushes their scores down further. That is a second, independent penalty on the same documents — see setting CJK chunk size in tokens.
• Unevenly translated documentation. A minority language version that is a partial translation contains fewer, older chunks. The ranking is then correct about relevance and wrong about what the user needed, which is harder to diagnose.
• Code-switched chunks. A chunk containing both languages sits between the two regions and ranks poorly for queries in either.

Four fixes, in order of effort

1. Tag and filter. Run language identification on every chunk at ingest, store it as metadata, detect the query language, and filter. This is an afternoon of work and removes the problem entirely for the common case where a user wants results in their own language. Its weakness is that it also removes the correct English answer when no Japanese document covers the topic.
2. Retrieve per language, then merge. Run top-k separately within each language partition and merge with reciprocal rank fusion, which combines by rank rather than by score and so is immune to the baseline offset between language regions. You get the best Japanese chunks and the best English ones, and the fusion decides. This is the default worth reaching for.
3. Boost by query language. Add a fixed score bonus to chunks matching the query language. Cheap, and it works, but the constant is a magic number that has to be retuned whenever the corpus balance or the embedding model changes.
4. Re-rank cross-lingually. Retrieve a wide candidate set — 100 or more, unfiltered — and pass it to a cross-encoder or an LLM re-ranker that scores query and passage jointly. Joint scoring does not have the language-region baseline problem because it is not comparing two independent vectors. This costs latency and money per query and is the most accurate of the four.

Fixes 2 and 4 compose well: fuse per-language retrievals into a candidate set, then re-rank it. Fixes 1 and 3 are alternatives to each other and should not both be applied.

Measuring whether you have this

You do not need a labelled evaluation set to detect this failure. Take a sample of real queries, group them by detected language, and compute the language distribution of the top-10 results for each group. Compare it to the corpus distribution.

If Japanese queries return 93% English results and the corpus is 93% English, the ranking is ignoring language entirely and you are seeing pure count dominance. If Japanese queries return, say, 60% English, the model is applying some language affinity and the count is still winning. Either way the gap between those two percentages is the size of the problem, and it is the number to watch after a fix.

The second measurement is the one that tells you whether a fix helped: for a set of queries where you know a correct minority-language chunk exists, record its rank before and after. A chunk moving from rank 40 to rank 3 is the whole point; a chunk moving from 40 to 25 means the boost constant is too small.