Eigenvectors, PCA, and Why Dimensionality Reduction Works

PCA finds the directions in which your data varies most, and lets you keep only those. The whole procedure is: centre the data, build the covariance matrix, take its eigenvectors. This page does that on five two-dimensional points where the answer comes out as 4.75 and 0.25, and then verifies both numbers a second way.

The problem PCA solves

A set of embeddings at 1,536 dimensions is expensive: 6 KB per vector in fp32, and every similarity search touches all 1,536 numbers. But the vectors do not fill that space. They lie close to a much lower-dimensional surface, because the model that produced them was itself constrained. PCA finds the best flat approximation to that surface, where “best” means preserving the most variance.

The reason variance is the right thing to preserve: variance is what distinguishes one vector from another. A direction in which every vector has the same value carries no information for telling them apart, and can be dropped at no cost.

The covariance matrix, computed

Five points, chosen so the arithmetic stays clean.

Data:      (1,2) (2,3) (3,5) (4,4) (5,6)

Means:     x_bar = (1+2+3+4+5)/5 = 3
           y_bar = (2+3+5+4+6)/5 = 4

Centred:   (-2,-2) (-1,-1) (0,1) (1,0) (2,2)

Centring is not optional. PCA measures variance about the mean, and skipping the subtraction makes the first component point at the mean instead of along the spread.

Covariance, dividing by (n-1) = 4:

Sxx = (-2)^2 + (-1)^2 + 0^2 + 1^2 + 2^2 = 10   ->  10/4 = 2.50
Syy = (-2)^2 + (-1)^2 + 1^2 + 0^2 + 2^2 = 10   ->  10/4 = 2.50
Sxy = (-2)(-2) + (-1)(-1) + (0)(1) + (1)(0) + (2)(2)
    = 4 + 1 + 0 + 0 + 4 = 9                    ->   9/4 = 2.25

C = [ 2.50  2.25 ]
    [ 2.25  2.50 ]

Read the matrix. The diagonal is the variance along each original axis, both 2.50. The off-diagonal is the covariance, 2.25, and it is large relative to the variances — the correlation is 2.25 / 2.50 = 0.9. The two coordinates are nearly redundant, which is exactly the situation PCA exists for.

Eigenvectors, and what they mean here

An eigenvector of a matrix is a direction the matrix does not rotate: C v = lambda v. It only stretches it, by the factor lambda, the eigenvalue. For a covariance matrix, those directions are the axes along which the data varies independently, and the eigenvalue is the variance along each.

A matrix of the form [[a, b], [b, a]] has an answer you can read off without solving anything:

C [1, 1] = [2.50 + 2.25, 2.25 + 2.50] = [4.75, 4.75] = 4.75 * [1,1]
C [1,-1] = [2.50 - 2.25, 2.25 - 2.50] = [0.25,-0.25] = 0.25 * [1,-1]

Eigenvalue 4.75, eigenvector [1, 1]  -> normalised [0.7071, 0.7071]
Eigenvalue 0.25, eigenvector [1,-1]  -> normalised [0.7071,-0.7071]

Check: eigenvalues must sum to the trace.
  4.75 + 0.25 = 5.00 = 2.50 + 2.50   correct.

Now the whole point of the exercise:

Total variance    = 4.75 + 0.25 = 5.00
Explained by PC1  = 4.75 / 5.00 = 95.0%
Explained by PC2  = 0.25 / 5.00 =  5.0%

Keeping only PC1 throws away one of two dimensions
and loses 5% of the variance.

Verifying the eigenvalues by projection

The claim is that the variance of the data along the first eigenvector is 4.75. Project the centred points onto it and take the variance, which is a completely independent route to the same number.

Project onto v1 = [1,1]/sqrt(2). The projection of a
centred point (a,b) is (a + b) / sqrt(2).

(-2,-2) -> -4/1.41421 = -2.82843
(-1,-1) -> -2/1.41421 = -1.41421
( 0, 1) ->  1/1.41421 =  0.70711
( 1, 0) ->  1/1.41421 =  0.70711
( 2, 2) ->  4/1.41421 =  2.82843
                          -------
mean                       0.00000

variance = (8 + 2 + 0.5 + 0.5 + 8) / 4 = 19 / 4 = 4.75

Same for v2 = [1,-1]/sqrt(2). Projection is (a - b)/sqrt(2).

(-2,-2) ->  0/1.41421 =  0.00000
(-1,-1) ->  0/1.41421 =  0.00000
( 0, 1) -> -1/1.41421 = -0.70711
( 1, 0) ->  1/1.41421 =  0.70711
( 2, 2) ->  0/1.41421 =  0.00000

variance = (0 + 0 + 0.5 + 0.5 + 0) / 4 = 1 / 4 = 0.25

Both eigenvalues confirmed by a route that never touched an eigen-decomposition. That is the definition doing what it says: the eigenvalue of a covariance matrix is the variance along its eigenvector.

Notice also that the projections onto v1 and v2 are uncorrelated — the first three points give (-2.83, 0), (-1.41, 0), (0.71, -0.71), and their covariance works out to zero. Decorrelating the coordinates is the other half of what PCA does, and it is why the components can be dropped independently.

Whitening, and why it usually hurts

A common next step after PCA is to divide each component by the square root of its eigenvalue, so every direction ends up with variance 1. That is whitening, and it is frequently suggested for embeddings on the grounds that it removes the dominance of a few directions. On the five-point dataset it does something clearly visible:

Eigenvalues: 4.75 and 0.25

Whitening divides PC1 by sqrt(4.75) = 2.1794
              and PC2 by sqrt(0.25) = 0.5000

Multipliers:  PC1  1/2.1794 = 0.4588
              PC2  1/0.5000 = 2.0000

Relative amplification of PC2 against PC1:
  2.0000 / 0.4588 = 4.359 = sqrt(4.75 / 0.25) = sqrt(19)

The direction carrying 5% of the variance is
amplified 4.36x relative to the one carrying 95%.

That amplification factor is sqrt(lambda_max / lambda_min), and in a real 1,536-dimensional embedding set the eigenvalue ratio between the first and last components is routinely four or more orders of magnitude. Whitening then amplifies the smallest directions by a factor of a hundred or more.

Those smallest directions are the ones estimated least reliably. Their eigenvalues are dominated by sampling noise, by numerical error in the decomposition, and by whatever quantisation the vectors have been through. Whitening takes the least trustworthy part of the decomposition and makes it as loud as the most trustworthy part, which is why it so often makes retrieval worse rather than better.

If you want to try it anyway, add a floor to the denominator — 1 / sqrt(lambda + eps) with eps a small fraction of the largest eigenvalue — and evaluate on retrieval quality rather than on how the variance spectrum looks afterwards. The spectrum will look better by construction; that is what the operation does, and it is not evidence.

Running it on your own embeddings

The procedure is identical at 1,536 dimensions; only the matrix size changes. This prints the cumulative variance curve, which is the number that decides how many dimensions you can afford to drop.

import numpy as np

# X: (n_vectors, n_dims), float32 or float64
X = np.load("embeddings.npy").astype(np.float64)

mean = X.mean(axis=0)
Xc = X - mean

# SVD rather than an explicit covariance matrix:
# same answer, better conditioned, and it does not
# build a 1536 x 1536 intermediate.
U, S, Vt = np.linalg.svd(Xc, full_matrices=False)

var = (S ** 2) / (len(X) - 1)
cum = np.cumsum(var) / var.sum()

for k in (8, 16, 32, 64, 128, 256, 512):
    if k <= len(var):
        print(f"{k:4d} dims -> {cum[k-1]*100:6.2f}% of variance")

# Project down to k dimensions
k = 128
Xk = Xc @ Vt[:k].T
print("reduced:", X.shape, "->", Xk.shape)

# Reconstruct and measure the actual error
Xr = Xk @ Vt[:k] + mean
err = np.linalg.norm(X - Xr) / np.linalg.norm(X)
print(f"relative reconstruction error: {err:.4f}")

1. Fit on a sample of at least a few thousand vectors from the same distribution as production. Ten thousand is plenty; the covariance structure converges quickly.
2. Save mean and Vt[:k]. Every future vector must be transformed with the same two arrays, or the reduced vectors are not in the same space and every similarity is meaningless.
3. Choose k from the cumulative variance curve, then verify on retrieval quality rather than on variance. 95% of variance retained does not guarantee 95% of recall.
4. Re-fit only when you re-embed everything. A changed projection invalidates every stored vector, exactly like changing the embedding model does.

Where PCA is the wrong tool

• Centring breaks cosine similarity. PCA subtracts the mean, and cosine similarity between mean-subtracted vectors is not the cosine similarity between the originals. If your index is built on cosine, either keep the mean-adding step in your transform or accept that you have changed the metric, and re-normalise after projecting.
• It is linear, and embedding spaces are not. PCA can only find flat subspaces. Where the structure is curved, a linear projection loses information that a trained reduction would keep — Matryoshka embeddings are the version of this problem solved at training time, and they beat PCA on the models that support them.
• Variance is not importance. The largest component of a set of text embeddings often encodes something dull — document length, or the general frequency direction — rather than anything you want to retrieve on. It is worth checking: remove the mean and the top component, re-run your retrieval evaluation, and see whether it improves. Sometimes it does.
• It does not reduce storage as much as it looks. Halving dimensions halves the bytes, but quantising from fp32 to int8 quarters them and usually costs less accuracy. The two compose, and quantisation is normally the first one to reach for.
• It needs the whole matrix in memory, or an incremental variant. The SVD above materialises the centred matrix. For tens of millions of vectors use a randomised or incremental PCA fitted on a sample.