Multilingual Search Without Separate Indexes

“One index for every language” is the right goal and the naive implementation of it is broken in a way that no relevance tuning will fix, because the damage is inside the scoring function’s own corpus statistics.

Three ways to be multilingual

The third is what most new systems reach for, and multilingual embeddings covers the model side of it. It does not remove the need for the lexical half — cross-lingual dense retrieval is weak on exactly the queries lexical matching is strong on, which is names, codes and product identifiers. So in practice you keep both, and the lexical half is where the interesting failure lives.

One index corrupts the statistics

BM25 depends on two corpus-level quantities: the document frequency of each term, which becomes IDF, and the mean document length avgdl, which normalises the length penalty. Both are computed over whatever is in the index. Put two languages in one field and both become meaningless.

Take the length term first, because it is the one nobody expects. Assume an index that is half English documents averaging 200 tokens and half Chinese documents averaging 90 tokens after segmentation. The shared avgdl is 145. With b = 0.75 and k1 = 1.2, for a term appearing twice:

length bracket = 1 - b + b * |d| / avgdl

ENGLISH document, |d| = 200, avgdl = 145
  200 / 145 = 1.3793
  bracket   = 0.25 + 0.75 * 1.3793 = 1.2845
  tf factor = 2 * 2.2 / (2 + 1.2 * 1.2845) = 4.4 / 3.5414 = 1.2425

CHINESE document, |d| = 90, avgdl = 145
  90 / 145  = 0.6207
  bracket   = 0.25 + 0.75 * 0.6207 = 0.7155
  tf factor = 2 * 2.2 / (2 + 1.2 * 0.7155) = 4.4 / 2.8586 = 1.5392

  1.5392 / 1.2425 = 1.239

A perfectly average Chinese document scores about 24% higher than a perfectly average English one, for the same term frequency, purely because its language tokenises into fewer units. Nothing about relevance changed. Compute avgdl per language and both documents get a bracket of exactly 1.0 and a factor of 1.375 — identical, which is correct.

IDF is corrupted the same way and in the opposite direction. A word that appears in every English document appears in half the mixed index, so ln(1 + (N - n + 0.5)/(n + 0.5)) gives it the IDF of a moderately selective term instead of nearly zero. Common function words in every language start carrying weight they should not have, and the effect grows with how unbalanced the language mix is.

Per-language fields

The fix is not separate indexes. It is separate fields inside one index: title_en, title_de, title_zh, each with its own analyzer. Lucene-derived engines compute document frequency and average length per field, so per-language fields give per-language statistics for free, while the documents stay in one place with one identifier and one set of filters.

A document populates only the fields for the languages it is written in; the rest are empty and contribute nothing. A query searches across the fields, either all of them or the subset that language detection suggests. The one thing you must not do is compare raw scores across fields — a BM25 score from the German field and one from the English field come from different distributions and are not on a common scale. Combine them by rank rather than by score, which is the same reciprocal rank fusion argument made in hybrid search for the same reason.

The analyzer traps

Each of these has cost somebody a week. They are language-specific and none of them is optional if you serve the language.

• Turkish dotted and dotless i. Turkish has two distinct letters where English has one, so lowercasing is locale-dependent: in a Turkish locale the capital I lowercases to the dotless form, not to the ASCII i. Apply a locale-sensitive lowercasing on one side of the pipeline and a locale-neutral one on the other and queries stop matching their own documents. The rule is that index-time and query-time normalisation must be byte-for-byte the same function, and the way to check is to run both over a fixture file in CI.
• German compounds. A single German noun can contain three searchable concepts, and a user searching for one of them matches nothing without decompounding. A dictionary-based decompounder is standard; the failure mode is over-splitting proper nouns into unrelated words.
• CJK segmentation. No whitespace, so tokenisation is a model and a different valid segmentation is a different query. Character bigrams are the cheap fallback: index overlapping pairs of characters and let the scoring sort it out. It inflates the index and it is far better than a wrong segmentation.
• Arabic and Hebrew. Optional diacritics that users omit and publishers include, plus clitics attached to words as prefixes. Normalisation has to strip diacritics on both sides and handle the prefixes, or half the corpus is unreachable.
• Accent folding as a one-way door. Folding é to e fixes French queries typed on an English keyboard and breaks any language where the accent is a distinct letter. Index both — a folded field for recall, an exact field boosted for precision — rather than choosing.

The detection problem you cannot solve

Everything above assumes you know which language a query is in. Document-side detection is reliable, because a document is hundreds of words. Query-side detection is not, because a query is two or three words and a large share of them are proper nouns that belong to no language at all. A brand name, a product code and a person’s name are the same string everywhere.

So do not build a pipeline that branches on a detected query language. Search all the language fields you plausibly serve, fuse by rank, and use signals you actually trust — the user’s interface locale, their country, their explicit preference — as a boost on the matching field rather than as a filter. Detection then becomes an optimisation that saves work when it is confident, instead of a single point of failure that silently returns nothing when a Dutch user searches for an English film title.