Multilingual Embedding Models for Local RAG

A multilingual embedding model is not an English model with extra training. Its shape is different, and the difference is concentrated in one tensor that turns out to be most of the model.

Where the extra parameters are

Compare two models of identical depth and width. bge-base-en-v1.5 uses a BERT backbone with a 30,522-token WordPiece vocabulary and is published at 110M parameters. multilingual-e5-base uses an XLM-RoBERTa backbone with a 250,002-token SentencePiece vocabulary and is published at 278M. Both are 12 layers at 768 dimensions with a 3,072-wide feed-forward block.

That entire 168M-parameter difference is the token embedding table, and the arithmetic reproduces both published totals. A transformer layer at width d with feed-forward width 4d holds four d x d attention projections and two d x 4d feed-forward matrices:

per layer  = 4d^2 + 8d^2 = 12d^2 = 12 x 768^2 = 7.08M
12 layers  = 84.9M                       (identical in both models)

embedding table = vocab x d
  BERT     30,522 x 768 =  23.4M   ->  84.9 + 23.4 = 108M   (published 110M)
  XLM-R   250,002 x 768 = 192.0M   ->  84.9 + 192.0 = 277M  (published 278M)

Both land within 2% of the published figure, and the residual is biases, layer norms and the pooler, which this arithmetic does not count. Vocabulary size for XLM-RoBERTa is taken from the FacebookAI/xlm-roberta-base configuration, which records vocab_size 250002 and hidden_size 768.

So: 21% of an English base model is its vocabulary table, and 69% of a multilingual one is. The part that does the thinking — the twelve transformer layers — is byte-for-byte the same size in both.

Heavy in RAM, not in compute

This is the fact with the most practical consequences, and it is counter-intuitive if you are used to reading parameter count as a proxy for cost.

An embedding table is not multiplied by anything. It is indexed: given a token id, read one row. The cost of that operation is one memory read per token, and it does not depend on how many rows the table has. So a 278M-parameter multilingual model performs almost exactly the same arithmetic per token as a 110M-parameter English one — the 84.9M of transformer weights are what get multiplied, and they are identical.

• Memory scales with the full count. 278M at fp32 is 1.11 GB against 440 MB, and every byte of the table must be resident because you cannot predict which rows a document will touch. On a constrained machine this is the number that decides, as the 8 GB arithmetic shows.
• Throughput scales with the transformer only. Expect a multilingual base model to embed at broadly the speed of an English base model, not at 40% of it. If you benchmark and find otherwise, the cause is tokenization producing more tokens per document, not the parameter count.
• Quantization pays differently. Quantizing the embedding table saves a great deal of memory and no compute, because the table is never in the arithmetic path. On a multilingual model that is 69% of the file for zero throughput risk, which makes it the first thing to quantize rather than the last.

Fertility, and the 512-token cap

A quarter-million-token vocabulary sounds generous until you divide it by a hundred languages. Each language gets a small share, and how large a share depends on how much of that language was in the corpus the tokenizer was fitted to. The measurable consequence is fertility: the average number of tokens a word costs.

For well-represented languages fertility is close to English’s. For under-represented ones, and especially for languages with rich morphology or a non-Latin script, the same sentence can cost substantially more tokens because it is being assembled from smaller pieces. You do not have to take this on faith — measure it on your own corpus:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("intfloat/multilingual-e5-base")

for lang, path in [("en", "sample.en.txt"), ("fi", "sample.fi.txt")]:
    text = open(path, encoding="utf-8").read()
    n_tok = len(tok.encode(text))
    n_word = len(text.split())
    print(lang, "tokens/word", round(n_tok / n_word, 2))

The number that comes out determines how much text fits in the model’s 512-token window. If one language runs at 1.3 tokens per word and another at 2.4, a chunking strategy expressed in words produces chunks of very different sizes in tokens — and the ones that exceed 512 are truncated silently. Chunk by tokens with the actual tokenizer, never by words or characters, or your non-English documents lose their tails.

The training mix decides the quality gap

Multilingual pre-training corpora are built by crawling, and the web is not evenly multilingual. The proportions in the corpus become the proportions of capacity the model spends, so retrieval quality varies across a model’s supported languages by much more than the word “supported” suggests. The multilingual-e5 model cards say this plainly: the models cover 100 languages, with degraded performance on low-resource ones.

There is a second, subtler effect specific to embeddings. What makes cross-lingual retrieval work — an English query finding a German document — is that the training pushed translations of the same content to the same region of the space. That alignment is itself learned from parallel or comparable data, and it is strongest between the language pairs that had the most of it. Cross-lingual retrieval between two well-served languages works well; between two under-served ones it can fail even when monolingual retrieval in each is acceptable.

The consequence for evaluation is that a single aggregate benchmark score is close to useless for this decision. What you need is the score for your languages, and if your languages are not on a public leaderboard, a hundred judged query-document pairs in each of them will tell you more than any published average.

What this means for a local index

• One model per index, always. Vectors from two different models are not comparable, so a mixed-language corpus needs one multilingual model over all of it, not an English model for the English documents and something else for the rest.
• Watch for clustering by language. A known failure mode of imperfectly aligned multilingual spaces is that documents group by language before they group by topic, so a query in one language retrieves only documents in that language. Test for it explicitly with a query whose best answer is in another language.
• Apply the prefix convention in every language. E5’s "query: " and "passage: " are the literal English strings regardless of the document’s language. Do not translate them; they are tokens the model was trained on, not instructions it reads.
• Budget for re-embedding. Multilingual model releases move faster than English ones, and the arithmetic above means each upgrade is a full corpus pass at roughly the cost of the English equivalent.