Quantising Vectors: Binary, Scalar and Product Quantisation

The compression numbers quoted in this field — 4×, 32×, 64× — are not benchmark results. They are ratios of bit widths, and you can derive every one of them in your head once you see where they come from.

The precision ladder

Each dimension of an embedding is stored in some number of bits. The entire compression story is choosing that number. For a 1536-dimension vector:

float32   32 bits/dim   1536 * 4      = 6,144 B    1x
float16   16 bits/dim   1536 * 2      = 3,072 B    2x
int8       8 bits/dim   1536 * 1      = 1,536 B    4x
binary     1 bit /dim   1536 / 8      =   192 B   32x
PQ, m=96   8 bits/sub   96 * 1        =    96 B   64x
PQ, m=48   8 bits/sub   48 * 1        =    48 B  128x

The famous 32× for binary is exactly 32 bits divided by 1 bit. The 64× for product quantisation with 96 subvectors is 6,144 bytes divided by 96. There is no measurement anywhere in that table, and any article presenting these ratios as findings has confused arithmetic with evidence.

What is not derivable is the recall cost, which depends on the model and the corpus. That is the part to test yourself, and the last two sections are about how the schemes differ in what they cost you.

One thing the table does make derivable is where the savings stop mattering. Going from float32 to int8 removes 4,608 bytes per vector; going from int8 all the way to product quantisation at 96 bytes removes a further 1,440. The first step is three quarters of the total available saving and it is by far the least damaging — which is the general shape of this ladder, and the reason the aggressive rungs should be justified by a memory ceiling you have actually hit rather than by the size of the ratio in the marketing.

Scalar quantisation

The simplest scheme: map each dimension’s float range onto 256 integer levels.

# fitted once on a sample of vectors, per dimension
lo = np.percentile(sample, 0.5, axis=0)      # not min(): outliers
hi = np.percentile(sample, 99.5, axis=0)

def q8(v):
    return np.clip(np.round((v - lo) / (hi - lo) * 255), 0, 255).astype(np.uint8)

Use percentiles rather than the true minimum and maximum. A single outlier vector stretches the range, and every other vector is then squeezed into a fraction of the 256 levels — a quiet, uniform loss of precision across the whole corpus caused by one row. Clipping the tail costs a little accuracy on the outliers and buys resolution for everything else.

int8 is the least dramatic and most reliable rung on the ladder. 4× less memory, distance computations that modern CPUs execute with dedicated integer instructions, and an error per dimension bounded by half a quantisation step. Embedding coordinates are roughly bell-shaped around zero with no meaningful structure at the eighth decimal place, which is why this works as well as it does.

Binary quantisation and Hamming distance

Keep only the sign of each coordinate. One bit per dimension, and the similarity function changes completely:

bits    = np.packbits(v > 0)          # 1536 floats -> 192 bytes
hamming = popcount(a XOR b)           # 1536 bits = 24 uint64 words

float32:  1536 multiply-adds  + 1536 * 4 bytes read
binary:     24 XOR + 24 POPCNT +  192     bytes read

Twenty-four machine instructions against fifteen hundred multiply-adds, over one thirty-second of the bytes. This is why binary quantisation is not merely a storage optimisation — it changes the shape of the whole system, converting a bandwidth-bound scan into something a single core can rip through.

What you have thrown away is magnitude. Two vectors pointing in slightly different directions but with the same sign pattern become indistinguishable, so the ordering within a group of similar documents is close to arbitrary. Binary quantisation is therefore a candidate generator, not a ranker, and using it as a ranker is the mistake that makes people conclude it does not work.

Rescoring, which is the whole trick

The pattern that makes aggressive quantisation viable is oversample and rescore, and it is what Cohere’s published guidance for its int8 and binary embeddings, the Faiss documentation for product quantisation, and the Qdrant and Sentence-Transformers documentation all describe. The shape is the same everywhere:

want top 10
  1. search the binary index for the top 200      (fast, in RAM, low precision)
  2. fetch those 200 full-precision vectors        (200 * 6144 B = 1.2 MB)
  3. rerank the 200 by exact float32 similarity
  4. return the top 10

memory held hot:  100e6 * 192 B      = 19.2 GB     (binary)
memory not held:  100e6 * 6144 B     = 614.4 GB    (float32, on SSD)
extra read per query:                  1.2 MB

A 614 GB problem becomes a 19 GB problem plus a megabyte of reads per query, and the final ordering is computed at full precision so it is identical to the exact answer whenever the shortlist contained the right documents. The oversampling factor — 20× in the example — is the dial: raise it until recall against an exhaustive baseline stops improving, then stop. Published figures for binary-plus-rescoring cluster in the range where most float32 recall is retained, but treat any specific percentage as corpus-dependent and measure it on yours against a brute-force ground truth. That measurement is an afternoon and it is the only number that binds you.

Product quantisation

Product quantisation (Jégou, Douze and Schmid, TPAMI 2011) predates the embedding era and is still the densest scheme in wide use. Split the vector into m contiguous subvectors, run k-means with 256 centroids over each subspace independently, and store m single-byte centroid ids:

d = 1536, m = 96  ->  each subvector is 16 dims
                      each subspace has a codebook of 256 centroids
                      stored code = 96 bytes

query time (asymmetric distance computation):
  build a 96 x 256 table of q-subvector-to-centroid distances   (24,576 ops)
  each candidate's distance = sum of 96 table lookups           (96 ops)

The asymmetric trick is the elegant part: the query is never quantised, so all the error is on the stored side, and after one table build each candidate costs 96 additions with no multiplications at all. The costs are a training pass over a representative sample, sensitivity to distribution shift after that training, and a reconstruction error large enough that PQ is nearly always paired with rescoring too.

Choosing a scheme

• Under 10 GB of vectors, do not quantise. You are adding an error source and a tuning parameter to save an amount of memory that does not change which machine you rent.
• float16 is the free one. Half the memory, error at the fourth decimal place, no training, no rescoring, no thought required. If you do exactly one thing from this page, do this.
• int8 when memory is the binding constraint. 4× with small and predictable loss, and fast integer distance kernels.
• binary plus rescoring above about 50 million vectors. The 32× is what makes very large indexes affordable, and rescoring is not optional.
• PQ when even binary is too big, or when you need a tunable ratio rather than the fixed 32×. Accept that you now own a trained artefact that can go stale.
• Combine with truncation deliberately, not accidentally. Truncating to 512 dimensions and then binarising is 6144/64 = 96×, and the two error sources compound. Choose the order, measure after each step.