Building a Multilingual Retrieval Index Without Separate Models Per Language

The per-language-model design sounds obviously better: a specialist for each language beats a generalist. Work the arithmetic and the storage argument disappears, the latency argument mostly disappears, and what is left is a scoring problem that has no clean solution.

Storage is not the differentiator

The first thing people compute is storage, and it is a wash. Your corpus has some number of chunks. Under one model they are one index; under per-language models they are the same chunks split across shards. The vectors are the same count.

One multilingual model, 1024 dims
  5,000,000 chunks * 1024 * 4 B                    = 20.5 GB

Six per-language models, 768 dims each
  5,000,000 chunks total, split across 6 shards
  5,000,000 * 768 * 4 B                            = 15.4 GB

Difference comes from the dimension, not from the sharding.
Per-shard HNSW graphs are slightly cheaper in total than one big
graph, because search cost is logarithmic in shard size — but the
bytes are dominated by vectors either way.

If anything, per-language shards look marginally better on storage, because smaller specialist models often use smaller dimensions. Nobody should choose an architecture on a fifteen percent storage difference. Set this argument aside; it is not the one that matters.

Latency, and the routing tax

Query-time cost splits into embedding the query and searching the index. Search is logarithmic in size for a graph index, so six shards of a sixth the size each are individually a little faster and collectively a little slower once you fan out. The interesting term is the routing.

• One model. One embedding call, one search. No language decision is required at any point, which is the quiet advantage.
• Per-language, with routing. Detect the query language, then one embedding call and one search. Detection with a compiled identifier is fast enough to ignore. The cost is not latency; it is that the detection can be wrong, and when it is, the query goes to a shard that cannot contain the answer and returns confident nonsense.
• Per-language, fanned out. If you cannot route reliably — and for two-word queries and code-switched text you cannot — you must embed the query with every model and search every shard. That is six embedding calls and six searches, so cost scales with the number of languages and latency is set by the slowest of them.

Routing reliability is the crux and it is worse than it looks. Language identification on a two-word query is close to a coin flip between related languages, and on romanized or code-switched input it is confidently wrong. Any design whose correctness depends on classifying short user text has a floor set by that classifier. See what to do when language detection fails.

The argument that actually decides it

Suppose you fan out and get results from six shards. Now merge them into one ranked list. You cannot, not by score.

A cosine similarity from model A and a cosine similarity from model B are numbers in the same range that mean different things. Each model’s space has its own scale and its own anisotropy: one may give unrelated text a baseline similarity of 0.3 and its best match 0.7, another a baseline of 0.7 and its best match 0.9. Merging by raw score hands every slot to whichever model has the more generous baseline, regardless of relevance. This is not a tuning problem; the two numbers are not on a common scale and no fixed transform makes them so, because the distributions differ per model, per language and per query.

Normalizing per shard — z-scoring against the shard’s own score distribution — helps and introduces its own distortion, because it assumes each shard contains a comparable number of relevant documents. For a query where the answer exists in one language only, that assumption is exactly false: the shard with no relevant documents still contributes its best scores after normalization, promoting irrelevant results into the top-k.

The clean answer is rank fusion, which discards the scores. Reciprocal rank fusion assigns each document a score of one over a constant plus its rank in each list, and sums across lists. It needs no calibration and no assumption about score distributions. It is also the reason the single-model design wins: with one model, ranks are directly comparable and you do not need fusion at all, so a whole class of tuning simply does not exist. And the single model can retrieve a German document for a French query, which no per-language architecture can do at all.

Building the single-model index

1. Normalize at ingest. NFC everywhere, plus the script-specific folding your languages need — Arabic letter unification and diacritic stripping, script variant selection for Chinese, zero-width character handling for Indic text. Store the normalized text for embedding and the original for display.
2. Detect and store the language per chunk. Not for routing — you are not routing — but as metadata. A chunk is long enough for reliable detection, unlike a query, and the label costs a string per row. Without it you cannot evaluate or filter per language later, and you will want to.
3. Chunk in tokens using the embedding model’s own tokenizer. Character-based chunking gives wildly different content volumes per language, and chunking with a different tokenizer than the model uses silently overflows its window for high-fertility languages.
4. Apply the model’s passage prefix when embedding chunks, and its query prefix when embedding queries. Asymmetric models lose real quality without this.
5. Store vector, language, script and source. Script as well as language, because a Hindi document in Latin letters and one in Devanagari behave differently and you will want to separate them when something goes wrong.
6. At query time, do not detect anything. Embed with the query prefix, search the whole index. Optionally boost the user’s interface language with a metadata filter rather than a score adjustment, so the behaviour is explicit.
7. Evaluate recall at ten per language using the stored labels, and watch the worst language rather than the mean. Build the probe sets from structure if the language has no benchmark — here is the method.

The escape hatch: one specialist shard

Sometimes one language really is unacceptable under the multilingual model, and a specialist exists that is much better. The move is to add it as an extra shard rather than to convert the architecture. Keep the single index for everything, add a second index for that one language only, query both when the query might be in that language, and merge by rank.

def rrf(result_lists, k=60, top_n=10):
    """Reciprocal rank fusion. Needs no score calibration, which is
    the whole point when the lists come from different models."""
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)[:top_n]

# main index always; the specialist shard when it might apply
lists = [main_index.search(q_vec, top=50)]
if maybe_language(query, "ta"):
    lists.append(tamil_index.search(ta_vec, top=50))

final = rrf(lists)

The constant k, conventionally 60, damps the influence of top ranks so a document must do well in more than one list to win outright. Note what this design preserves: the main index still covers the language, so a wrong guess by maybe_language costs you the specialist’s contribution and nothing else. Compare that with routing, where a wrong guess sends the query to a shard that cannot answer it. Additive architectures degrade; exclusive ones fail.