MTEB and How Embedding Models Are Ranked
9 min read · updated August 4, 2026
MTEB is not one benchmark. It is a few dozen datasets across eight task families, each scored with the metric conventional for its family, and then averaged. Since accuracy, v-measure, Spearman correlation and nDCG@10 are not the same kind of number, the average at the top of the leaderboard is a mean of things that are not commensurable — and reading it as a quality score is the mistake the leaderboard invites.
What MTEB is
The Massive Text Embedding Benchmark was introduced by Muennighoff and colleagues in 2022. Its motivation was reasonable and it worked: before it, embedding models were evaluated almost entirely on semantic textual similarity, which is one narrow use of an embedding, and models tuned for it were being adopted for retrieval where they underperformed.
MTEB fixed that by evaluating one frozen embedding model across many task types at once. Nothing is fine-tuned per task — the model produces vectors, and each task defines a procedure that turns vectors into a score. The original release covered dozens of datasets across more than a hundred languages; the multilingual successor (MMTEB) expanded it to hundreds of tasks and languages with per-language views. Check the current leaderboard for which version a number came from, because the averages are not comparable across versions.
The task types, and the metric for each
| Task family | Description |
|---|---|
| Retrieval | Embed a corpus and a set of queries, rank by cosine similarity, score with nDCG@10. Largely drawn from BEIR. This family has the most datasets, which matters for the average. |
| Reranking | Given a query and a candidate list, order it. Scored with mean average precision (MAP) or a similar rank metric. |
| Classification | Embed the texts, fit a logistic regression probe on the training embeddings, score accuracy or F1 on the test embeddings. The probe is part of the evaluation, so this measures linear separability of the embedding space, not the model's own judgement. |
| Clustering | Embed, run k-means, compare the clusters to the gold labels with v-measure. Sensitive to the number of clusters chosen and to random initialisation. |
| Pair classification | Decide whether two texts are duplicates or entailed. Scored with average precision over the similarity threshold. |
| STS | Semantic textual similarity: correlate cosine similarity with human similarity ratings. Scored with Spearman rank correlation, which lives on a different scale from every accuracy above it. |
| Summarisation | Correlate embedding similarity between a machine summary and human summaries with human quality ratings. Also a correlation. |
| Bitext mining | Match sentences to their translations across languages. Scored with F1 over the matching. |
Two of these are correlations, one is an information-theoretic cluster agreement measure, one is a ranking measure with a cut-off at 10, and the rest are accuracies. They have different floors, different ceilings, and different sensitivities. Spearman on an STS set can be negative. Accuracy on a binary classification set floors at 50 for a balanced set. nDCG@10 is bounded by whether the relevant documents were judged at all — the subject of BEIR.
Why the headline average is not a score
The leaderboard’s primary column is the mean of the per-dataset scores. Three things follow.
- Task families are weighted by dataset count, not by importance. Retrieval contributes many datasets and STS contributes few, so the average is dominated by retrieval whether or not that is what you want. If you are building a classifier, the number at the top is mostly about a different job.
- Differently scaled numbers are added. A five-point gain in Spearman correlation and a five-point gain in nDCG@10 are not equivalent quantities of improvement, and averaging them asserts that they are.
- The average compresses variance. Two models with the same mean can have completely different profiles — one strong at retrieval and weak at clustering, the other the reverse. The mean hides exactly the information you needed.
The correct use of MTEB is to filter to the task family that matches your job and read that column, then look at the individual datasets inside it for ones whose domain resembles yours. If you do retrieval over support tickets, the FiQA and CQADupStack-style datasets tell you more than the mean of fifty-six numbers does.
The columns that actually decide the choice
Three properties beside the score usually decide which embedding model you can ship, and all three are on the leaderboard where nobody looks.
| Column | Description |
|---|---|
| Embedding dimension | Drives storage and index cost linearly. 4,096 dimensions at float32 is 16 KB per vector: ten million chunks is 160 GB before the index. A 768-dimension model at the same quality is a sixth of that. Some models support Matryoshka truncation, which lets you cut dimensions with a graceful quality loss instead of a cliff. |
| Max sequence length | Text beyond this is truncated silently by most client libraries. A 512-token limit means your 1,000-token chunks are half-embedded and you will not be told. This decides your chunking strategy before quality does. |
| Parameters and licence | Whether you can self-host it, what it costs to run, and how fast a batch embeds. A large model that must be called over an API has a different operational shape from a 100M-parameter one you run next to your database. |
There is also an asymmetry that the leaderboard does not surface: many models require different prefixes for queries and for documents, and using the wrong one degrades retrieval quietly. That is covered in embedding text versus embedding questions, and it is a more common cause of bad retrieval than model choice is.
The test sets are public and the training sets are not
Every MTEB dataset is public, including its test split. Most embedding models publish a leaderboard position and do not publish their training corpus. Those two facts together mean you cannot verify, from outside, that a model was not trained on the training splits of the datasets it is being evaluated on — and training on the train split of a public retrieval dataset is entirely legitimate practice, which makes the resulting score much less informative about held-out behaviour.
The visible symptom is a model that ranks high on MTEB and performs ordinarily on your corpus. That is the expected outcome when a benchmark is public, valuable and old enough to have been optimised against — the general form of the argument is in benchmark contamination, and it applies with particular force here because the incentive is concentrated: an embedding model’s entire market position is one leaderboard row.
What it is a bad proxy for
- End-to-end RAG quality. Retrieval is one stage. A better embedding model can be neutralised by bad chunking, and a mediocre one can be rescued by a cross-encoder rerank or by adding BM25. Evaluate the pipeline, not the component — retrieval and generation separately.
- Your domain vocabulary. MTEB datasets are largely web, Wikipedia, scientific abstracts and forum text. Internal jargon, product codes and abbreviations are exactly where general embedding models are weakest and where the benchmark is silent.
- Languages outside the evaluated set. A multilingual average across a hundred languages says nothing about the one you need. Read the per-language view — cross-language retrieval covers where the gaps fall.
- Late-interaction and multi-vector approaches. The benchmark assumes one vector per text. Architectures that keep a vector per token are scored under an assumption that does not fit them — see sparse, dense and late interaction.