Choosing Embedding Dimensionality for a Multilingual Index

The dimension question is usually posed as storage against recall. For a multilingual index there is a third term, and it decides the answer: the corpus is not the same size in every language, and the dimensions you would truncate away are not carrying the same thing for every language.

The storage arithmetic

The raw vector storage is one multiplication, and it is worth writing out because the number is often smaller than people expect and the index overhead is often larger.

raw_bytes = n_vectors * dims * bytes_per_component

5,000,000 chunks * 1024 dims * 4 bytes (float32)  = 20.5 GB
5,000,000 chunks *  768 dims * 4 bytes            = 15.4 GB
5,000,000 chunks * 3072 dims * 4 bytes            = 61.4 GB

HNSW adds a graph: roughly M * 2 * 4 bytes of links per vector,
plus the vectors themselves held in RAM for distance computation.
At M = 32 that is ~256 bytes/vector of links — small next to a
4 KB float32 vector at 1024 dims, and large next to a 128-byte
binary-quantized one.

Two things fall out. Dimension is a linear term, so halving it halves storage and roughly halves the distance-computation cost per candidate. And at high dimensions the vectors dominate the graph, so index overhead is a rounding error — which stops being true the moment you quantize, at which point the graph becomes the larger half.

Your chunk count is not language-independent

Here is the term the single-language version of this calculation does not have. If you take one corpus and publish it in twelve languages, you do not get twelve times the chunks. You get a different number per language, because chunking is done in tokens and tokenizer fertility varies by several times across languages.

ASSUMPTIONS (measure fertility with your own model's tokenizer)
  source content   = 40,000,000 words per language
  chunk size       = 512 tokens
  fertility (tokens/word), assumed:
      English  1.3    German  1.6    Hindi   3.0
      Arabic   2.6    Thai    3.4    Yoruba  4.0

DERIVED chunks per language = words * fertility / 512
  English  101,600      German  125,000     Hindi   234,400
  Arabic   203,100      Thai    265,600     Yoruba  312,500

Same content. Yoruba produces ~3x the vectors English does.

That changes the budget and it changes something more important. Each chunk in a high-fertility language contains roughly a third as many words as an English chunk of the same token size, so it carries less context, is more likely to split a sentence or an idea, and produces a less distinctive vector. The languages that generate the most storage are the ones whose chunks are individually least useful. If you are going to spend budget somewhere, spending it on a larger token chunk size for high-fertility languages is a better trade than spending it on extra dimensions everywhere.

What truncation removes first

Several current embedding models are trained with Matryoshka representation learning, introduced by Kusupati and colleagues in 2022 in “Matryoshka Representation Learning”. The training objective makes every prefix of the vector a usable embedding on its own, so you can keep the first 256 of 1024 dimensions and still retrieve, with graceful rather than catastrophic degradation. That is a genuinely useful property and it is why the dimension decision is now a runtime knob rather than a model choice.

The multilingual caveat is a consequence of how those leading dimensions get used. The earliest dimensions carry the directions the training objective found most globally useful, and in a multilingual space one of the highest-variance directions is language identity — the thing that makes embeddings cluster by language before they cluster by meaning. So there is a reasonable mechanistic expectation that truncation preserves the coarse structure, including language separation, and discards finer semantic distinctions first, and that it discards them fastest in the languages whose fine structure was least well trained.

That is a prediction from mechanism, not a measurement, and it is exactly the sort of claim you should not take on faith. It is also cheap to test: embed a few hundred items per language once at full dimension, truncate and renormalize in memory at 768, 512, 256 and 128, and compute recall at ten per language at each width. You get a curve per language from a single embedding pass. The languages whose curves fall off first are the ones that set your floor.

Quantization is usually the better lever

If the problem is storage rather than latency, dimension is the wrong knob to reach for first. Reducing the bytes per component gives a larger factor for less loss.

• float32 to int8 — four times smaller, per-dimension scaling, and typically a small recall cost. This is close to free.
• Binary — one bit per dimension, thirty-two times smaller, ranking by Hamming distance. Recall drops noticeably, and the standard repair is to over-retrieve with the binary index and rescore the top few hundred candidates with full-precision vectors kept on disk.
• Truncate and quantize together — halving dimensions and going to int8 is an eight-fold reduction, which for most indexes is the whole problem solved.

Quantization interacts with the multilingual case the same way truncation does: it compresses the space, and a language whose items are already crowded into a small region loses relatively more. Measure per language, not in aggregate.

How to decide, and what to measure

The failure that makes all of this moot is measuring recall as one number. An aggregate is weighted by query volume, so it is dominated by whichever language your existing users already speak — which is the language that already works. A change that improves the aggregate while destroying retrieval in your third-largest market looks like a win in the dashboard.

1. Tag every chunk with its language at ingest, even if you use one model for everything. It costs a string per row and it is the only way to slice any metric afterwards.
2. Build a labelled probe set per language, a hundred or so query-answer pairs. For a language with no benchmark, generate it from structure: building a test set from scratch.
3. Embed once at full dimension, then sweep width and quantization in memory. Report recall at ten per language and the worst language, not the mean.
4. Pick the smallest configuration where the worst language is still acceptable, and spend the savings on chunk size for the high-fertility languages.