Choosing an Embedding Model: Dimensions, Cost and Quality
5 min read · updated August 3, 2026
“Best embedding model” is a question with a leaderboard answer and a correct answer, and they are rarely the same model. The leaderboard optimises average score across dozens of academic tasks. You have one task, a licence constraint, and a bill that arrives every time you change your mind.
The six things that actually differ
| Property | Description |
|---|---|
| dimension | 384 to 4096 in common models. Sets storage and index memory linearly. A 3072-dim model costs exactly 8× a 384-dim one to keep in RAM. |
| max input | 512 tokens on most BERT-derived open models, 8,191 on the OpenAI text-embedding-3 family. This dictates your chunk size, not the other way round. |
| price / hosting | Per input token if hosted; a GPU or a CPU box if self-run. A 33M-parameter model like bge-small runs acceptably on CPU; a 7B embedding model does not. |
| prefix convention | E5 wants "query: " and "passage: "; BGE wants an instruction on queries only; Cohere takes an input_type parameter. Getting this wrong costs real recall silently. |
| licence | Several strong open models ship under non-commercial terms. Check before you build on one; this is the single most common late-stage surprise. |
| truncatability | Whether the model was trained with Matryoshka representation learning, i.e. whether you may cut the vector short later. This is an option on your future storage bill. |
Note what is not on that list: benchmark rank. It is on the list below, demoted.
Why MTEB rank is the weakest signal
MTEB — the Massive Text Embedding Benchmark, Muennighoff et al., 2022 — is a genuinely useful piece of work. It aggregates dozens of datasets across retrieval, clustering, classification, reranking, semantic similarity and more, and it made the field comparable. It is also, by now, the thing everyone trains against, and a benchmark that everyone trains against stops being a held-out test.
Two structural problems with reading the top of the table. First, the headline number is an average across task types, and you do not have an average task. A model that wins on classification and semantic textual similarity can be mid-table on retrieval, which is probably the only column you care about. Read the retrieval sub-score, and inside it the datasets that resemble your domain.
Second, the zero-shot picture is less flattering than the leaderboard suggests. BEIR (Thakur et al., 2021), the retrieval benchmark MTEB absorbed, was built precisely to test out-of-domain generalisation, and its headline finding was that BM25 remains a strong baseline that many dense retrievers fail to beat on unseen domains. That finding is what you should carry into your own evaluation: a model’s score on eighteen public corpora predicts its behaviour on your support tickets only loosely.
The switching cost, priced
Embeddings from two different models are not comparable. Not approximately comparable — meaningless together, because each model learned its own arbitrary axes. Changing models means re-embedding the entire corpus, so the choice carries a bill you can compute up front.
Take a corpus of 5 million chunks averaging 350 tokens. That is 1.75 billion input tokens. At OpenAI’s published January 2024 launch rate of $0.02 per million tokens for text-embedding-3-small, the full pass is $35. At $0.13 per million for text-embedding-3-large, it is $227.50. Both numbers are small, and that is the point worth internalising: at this scale the model price is nearly irrelevant to the decision, and choosing the cheaper model to save $190 is optimising the wrong line item. Storage and index memory, which scale with dimension, will cost more than that every month.
The arithmetic changes character at 500 million chunks, where the same calculation gives 175 billion tokens, $3,500 and $22,750. There the model price is a real decision, and so is the fact that you will pay it again the day you switch.
A selection procedure that ends in a number
Three shortlisted models, one afternoon, and an answer that is about your data rather than about Reddit’s.
- Build a gold set of 50 queries. Take them from your real search logs or support inbox, not from your imagination. For each, mark the chunk or chunks that genuinely answer it. Fifty is enough to separate models that differ; it is not enough to rank two that are close, and knowing that saves you from over-reading a two-point gap.
- Embed the corpus with each candidate — with each model’s own prefix convention applied correctly, which is where most home-grown comparisons go wrong.
- Score recall@10 and MRR@10. Recall@10 asks whether the right chunk is anywhere in the ten you will feed the model. That is the number a RAG system actually lives on; nDCG differences in the third decimal are not.
- Then apply the constraints — licence, max input, dimension against your storage budget, whether it can be truncated. A model that wins by two points and is 4× the storage is usually the wrong answer.
Keep the gold set in version control. It is the asset, not the model choice; it is what lets you evaluate the next candidate in an hour instead of an afternoon, and it is what tells you whether a vendor upgrade helped.
Reasonable defaults if you must guess
Structural advice, deliberately not a ranking. If you want a hosted model and English content, a mid-sized 1536-dimension general model from a major provider is a defensible default that nobody gets fired for; the text-embedding-3 family additionally supports truncation via a dimensions parameter, which preserves your options. If you need on-premises or per-request cost near zero, the BGE and E5 families at 384 or 768 dimensions run on CPU and are permissively licensed. If your content is multilingual, that constraint outranks everything else on this page and you should start from models that name their language coverage explicitly.
Whatever you pick, record the model name and version alongside every vector you store. The alternative — an index containing vectors from two model versions, indistinguishable, silently producing nonsense for a subset of queries — is the worst failure mode in this cluster and it is entirely preventable with one column.