Embedding Dimensions: Does 3072 Beat 768?

Dimension is the one model property whose cost you can compute to the byte before you commit to anything. The quality side cannot be computed — but vendors have published enough about their own models that you do not have to guess at that either.

What a dimension costs, exactly

A float32 is four bytes, so a d-dimensional vector is 4d bytes and the whole relationship is linear with no constant term worth arguing about. For a corpus of 10 million chunks:

  384 dims:  10e6 * 384  * 4 =  15,360,000,000 B =  15.4 GB
  768 dims:  10e6 * 768  * 4 =  30,720,000,000 B =  30.7 GB
 1536 dims:  10e6 * 1536 * 4 =  61,440,000,000 B =  61.4 GB
 3072 dims:  10e6 * 3072 * 4 = 122,880,000,000 B = 122.9 GB

Add the index. An HNSW graph with M = 16 stores up to 2M = 32 neighbour ids at the bottom layer, four bytes each, so roughly 128 bytes per vector plus a small amount for the sparse upper layers. For 10 million vectors that is about 1.3 GB — independent of dimension. Which means the graph is 8% of the bill at 384 dimensions and 1% at 3072. People tune M to save memory and are almost always tuning the wrong term.

Two secondary costs scale with the same factor and are worth naming. Every network response carrying a vector is 4d bytes of JSON — considerably more, in fact, since a float serialised as text is typically a dozen characters rather than four bytes, which is why bulk embedding endpoints and binary formats exist. And every backup, every replica and every cross-region copy multiplies the same figure.

The step from 61.4 GB to 122.9 GB is the interesting one, because it is the step from “fits in a 128 GB machine with room for Postgres and the operating system” to “does not”. Dimension choice is usually not a smooth cost curve; it is a discrete decision about which machine you are renting.

What the published numbers say

Nobody here has benchmarked these models, so the honest source is what the vendors published. When OpenAI announced the text-embedding-3 family in January 2024 it gave figures for its own models on two public benchmarks — MTEB for English tasks and MIRACL for multilingual retrieval:

Read the middle row first. text-embedding-3-small and ada-002 have identical dimension and differ by 1.3 MTEB points and 12.6 MIRACL points. That difference is entirely training, not width. Dimension is a capacity ceiling; it is not the thing that made the model better.

The same announcement stated that text-embedding-3-large truncated to 256 dimensions still scores above unshortened ada-002 at 1536 on MTEB. A vector one sixth the size beating the previous generation outright is the clearest available evidence that dimension count is a poor proxy for quality across model generations — and it is the reason truncatable embeddings are worth understanding before you pick a width.

Why returns diminish

There is an intuition behind the flattening. An embedding is a compression of a distribution of texts, and the information in that distribution has an effective rank — a number of genuinely independent directions that the data varies along. Add dimensions beyond that and the model has capacity to spare, which it spends on progressively less discriminative structure. Run principal component analysis over a sample of your own vectors and you can see it directly: the cumulative explained variance curve typically reaches most of its total in a small fraction of the dimensions. That is a diagnostic you can run in ten lines of numpy on your own data, and unlike a leaderboard it is about your corpus.

The second effect works against high dimension. As dimension grows, the distances between random points concentrate — the ratio between the nearest and farthest neighbour of a query tends toward one. This is the familiar curse of dimensionality, and its practical consequence is that approximate indexes have less contrast to work with, so achieving a given recall costs more graph traversal. Wider vectors are not just bigger; they are marginally harder to index.

Dimension and query time

Query work is linear in dimension too, and you can put an order of magnitude on it. An HNSW search with ef_search = 100 and M = 16 evaluates on the order of a few thousand candidate distances — take 1,600 as a working figure. Each distance reads one stored vector, 4d bytes, and the traversal is a random-access pattern, so this is bandwidth-bound rather than arithmetic-bound:

bytes touched per query = 1600 * 4 * d

  d =  768:  1600 * 3072  =  4.9 MB
  d = 1536:  1600 * 6144  =  9.8 MB
  d = 3072:  1600 * 12288 = 19.7 MB

at an effective 20 GB/s of random-access memory bandwidth:
  0.25 ms  vs  0.49 ms  vs  0.98 ms

Those figures are an order-of-magnitude derivation from the parameters, not a benchmark of any real system — cache behaviour, SIMD width and how the graph is laid out in memory all move them. The shape is what matters: sub-millisecond at every width for a single query, which is why nobody notices dimension in a demo, and 4× the memory traffic between 768 and 3072, which is exactly what you notice at a thousand queries per second on one box.

A decision rule

• Under a million vectors, ignore this page. A million 3072-dim vectors is 12.3 GB. Take the best model and move on.
• Between 1M and 50M, pick the model first and the width second. Prefer a model that supports truncation, so width stays a decision you can revisit without re-embedding.
• Above 50M, width is a budget line. 3072 dimensions at 100M vectors is 1.23 TB before the index. At that scale the question is not 768 versus 3072 but which combination of truncation and quantisation gets you inside a machine you can afford.
• Never compare widths across model families. A 768-dim model from this year beating a 1536-dim model from two years ago is the normal case, not an upset.