Evaluating Embedding Quality for a Language With No Benchmark

Your language is not on MTEB, the model cards do not mention it, and you still have to pick an embedding model this week. Two hundred labelled pairs from your own corpus will answer the question better than any leaderboard would have, and you can build them without an annotator.

What you are actually measuring

Be precise about the question, because the wrong question is easy to answer and useless. You are not measuring whether the model “understands” the language. You are measuring whether, for your content, the vector of a realistic query is closer to the vector of the right passage than to the vectors of the wrong ones.

That framing has a consequence: the test must be a ranking task over a realistic pool, not a similarity score on isolated pairs. A model can give every pair of sentences in a low-resource language a cosine of 0.85 and still rank correctly, and a different model can give the true pair 0.92 while giving fifty distractors 0.95. Absolute similarity numbers tell you nothing across models or across languages. Ranks do.

Three things are worth measuring separately, because they fail separately: whether the model can find a paraphrase, whether it can distinguish two passages that differ in one crucial detail, and whether it can retrieve across languages if you need that.

Getting positive pairs for free

The expensive way to build a test set is to pay bilingual annotators to write queries. The cheap way is to notice that your corpus already contains thousands of labelled pairs, created as a side effect of how documents are structured. Every one of these is a pair where you know the answer without anyone judging anything:

• Title and body. A document title is a human-written short description of its own content. Title as query, first paragraph as target. This is the single richest source and it exists in every corpus.
• FAQ question and answer. Real user phrasing paired with the passage that answers it. If you have a help centre, you have a retrieval test set.
• Support ticket subject and body. Written by users, in their own register, which is what your production queries look like.
• Summary and full text. Abstracts, meta descriptions, list-page blurbs, release-note headlines.
• Revisions of the same page. Two versions of a document from a version history are a near-paraphrase pair, and they test robustness to wording change specifically.
• Translations, if you have them. Any localised page paired with its source is a cross-lingual probe, and localisation files are the most under-used evaluation data in most companies.

Aim for around two hundred pairs. Below about a hundred the confidence interval on recall at one is wide enough to hide a real difference between two models; above a few hundred you are spending time for precision the decision does not need. Sample across your document types rather than taking the first two hundred rows, since a set drawn entirely from one section measures that section.

Generating hard negatives mechanically

A test where the distractors are random documents is too easy — nearly any model passes it, so it does not discriminate. What discriminates is near-misses, and you can manufacture those from the positives with string edits that need no language knowledge.

• Change a number. Take the target passage and alter a quantity, a version, a price or a date. A model that ranks the altered copy above the original for a query mentioning the real number will do the same thing to your users.
• Swap a named entity. Replace a product, place or person with another from your own corpus. Entities are often the only thing distinguishing two otherwise identical support articles.
• Negate. Insert or remove the language’s negation particle. Embedding models are famously weak at negation, and you want to know how weak before you rely on one.
• Change a unit or a direction. Import against export, enable against disable, before against after.
• Vary the surface form. For a morphologically rich language, re-inflect the query — nominative to genitive — and check that the ranking survives. For a language with diacritics, strip them. For a language with two scripts, transliterate.

The last one is really a robustness test rather than a negative, and it is the one most likely to change your architecture, because a big drop there means you need a normalization pass rather than a different model.

The metrics, and a collapse diagnostic

Rank each query against a pool containing its true target plus every other target plus the generated negatives. Report recall at one and mean reciprocal rank. Then run one extra diagnostic that is specific to the multilingual case and takes four lines.

import numpy as np

def unit(v):
    return v / np.linalg.norm(v, axis=-1, keepdims=True)

Q = unit(np.array([embed(q) for q in queries]))    # n x d
D = unit(np.array([embed(d) for d in targets]))    # n x d  (aligned order)

sims  = Q @ D.T
gold  = np.diag(sims)
ranks = (sims > gold[:, None]).sum(axis=1)          # 0 means rank 1

print("recall@1 ", float((ranks == 0).mean()))
print("recall@10", float((ranks < 10).mean()))
print("MRR      ", float((1.0 / (ranks + 1)).mean()))

# Language-collapse diagnostic: are unrelated same-language pairs
# scoring as high as genuinely related pairs? If so, the space is
# organised by language and your recall number is luck.
rand_same  = sims[np.random.randint(0, len(Q), 2000),
                  np.random.randint(0, len(D), 2000)].mean()
print("gold mean", float(gold.mean()), "random same-language mean", float(rand_same))

Read the last line carefully. If the gold mean and the random same-language mean are close, the model is putting everything in this language into one small region, and its apparent similarity scores are measuring language rather than meaning. That is a different failure from low recall and it has a different fix — see why embeddings cluster by language.

One more number, computed before you embed anything: tokenizer fertility. Tokenize a hundred words and divide. It predicts cost and chunking behaviour directly and is a reasonable leading indicator of how much of the language the model saw in training. It is not a quality measurement and should not be reported as one, but a fertility of five tokens per word tells you what to expect from the rest of the run.

Running the comparison

1. Assemble 200 query-target pairs from the structural sources above. Store them as a two-column file, and keep the document type in a third column so you can slice results later.
2. Generate one hard negative per positive with the edit rules. You now have a pool of about 400 targets, which is a realistic ranking difficulty for a set this size.
3. Normalize both sides identically — NFC at minimum, plus whatever script-specific folding your language needs. Do this before embedding, and use the same function in production or the evaluation is measuring a pipeline you will not ship.
4. Embed with each candidate model, applying that model’s required query and passage prefixes. Several models expect asymmetric prefixes and are noticeably worse without them; omitting them is the most common way an evaluation slanders a model.
5. Compute recall at one, recall at ten, MRR and the collapse diagnostic for every model. Record fertility and the cost of the embedding pass alongside them.
6. Repeat on the robustness variants — re-inflected, diacritic-stripped, transliterated queries. The gap between the base run and these is your normalization to-do list.
7. Pick on recall at ten and cost, not on the highest recall at one. Top-k retrieval feeding a language model cares about the answer being in the window, not about it being first.