Vector Norms, and Why Normalising Embeddings Matters

Normalising an embedding divides it by its own length so it becomes a pure direction. Do it and inner-product search behaves identically to cosine similarity; skip it and long vectors win regardless of what they mean, which produces plausible-looking results that are systematically wrong.

Three norms on one vector

A norm is a measure of length. There is more than one, and they disagree. Take v = [3, 4]:

L2  (Euclidean)  ||v||_2   = sqrt(3^2 + 4^2) = sqrt(25) = 5.0
L1  (Manhattan)  ||v||_1   = |3| + |4|             = 7.0
Linf (max)       ||v||_inf = max(|3|, |4|)         = 4.0

In general, the p-norm:
  ||v||_p = ( sum_i |v_i|^p ) ^ (1/p)

  p = 1    sum of absolute values
  p = 2    the ordinary length
  p -> inf the largest absolute component

L2 is the one that matters for embeddings, because cosine similarity, Euclidean distance and the dot product are all defined in terms of it. L1 appears in regularisation (where it drives weights to exactly zero in a way L2 does not) and occasionally in sparse retrieval.

Normalising, and what it throws away

v_hat = v / ||v||_2

[3, 4] / 5 = [0.6, 0.8]

Check: 0.6^2 + 0.8^2 = 0.36 + 0.64 = 1.00

The same vector under the other norms:
  L1:   [3, 4] / 7 = [0.428571, 0.571429]   sums to 1
  Linf: [3, 4] / 4 = [0.75, 1.0]            max is 1

Three different unit vectors. "Normalised" without
naming the norm is ambiguous; for embeddings it always
means L2.

Normalising discards magnitude and keeps direction. For text embeddings that is almost always right, because magnitude tends to encode things you do not want to rank on — text length, token count, how many sub-embeddings were pooled together. Two documents saying the same thing at different lengths should be equally similar to a query, and normalisation is what makes them so.

It is also idempotent, so normalising twice is harmless and there is no cost to being defensive about it.

The retrieval bug, and the one-line fix

Here is the failure in full. A vector index configured for inner product, storing vectors that were never normalised.

Query   q  = [1.00, 0.00]     (unit)

Stored:
  d1 = [0.60, 0.80]    ||d1|| = 1.00    angle 53.1 deg
  d2 = [0.99, 0.14]    ||d2|| = 1.00    angle  8.0 deg
  d3 = [2.00, 1.50]    ||d3|| = 2.50    angle 36.9 deg

Inner product:                Cosine:
  d3  2.00   <- top             d2  0.99   <- top
  d2  0.99                      d3  0.80
  d1  0.60                      d1  0.60

Different top result. d3 wins on inner product only
because it is 2.5x longer, while pointing 37 degrees
away from a query that d2 matches to within 8 degrees.

The fix:

import numpy as np

def l2_normalize(X, eps=1e-12):
    X = np.asarray(X, dtype=np.float32)
    norms = np.linalg.norm(X, axis=-1, keepdims=True)
    return X / np.maximum(norms, eps)

# Before insertion AND before every query.
vectors = l2_normalize(vectors)
query   = l2_normalize(query)

# d3 becomes [0.8, 0.6], inner product 0.80,
# and the ranking now matches cosine exactly.

The np.maximum(norms, eps) is not decoration. A zero vector — from an empty string, a failed API call that returned zeros, a padding row — has norm 0, and dividing by it gives NaN. One NaN in a similarity computation propagates to the whole score array and the query returns nothing, or returns arbitrary results, depending on how the index handles it.

Why this bug survives review: the unnormalised results are not obviously broken. They are relevant documents, just systematically the longer ones. Recall drops a few points, nobody sees an error, and the usual conclusion is that the embedding model is mediocre. The check is the norms, and the identity that makes the fix work is two lines of algebra.

Three places the norm quietly stops being 1

Mean pooling

Token vectors:  [1, 0], [0, 1], [1, 1]

mean = [(1+0+1)/3, (0+1+1)/3] = [0.6667, 0.6667]
||mean|| = sqrt(0.4444 + 0.4444) = 0.9428

Not 1. Normalise after pooling, always.

Any pooling operation — mean over tokens, mean over chunks, the centroid of a cluster — produces a vector whose norm depends on how much the inputs agreed with each other. Vectors pointing in similar directions average to something near unit length; vectors pointing in different directions average to something much shorter. That length is then read by an inner-product index as relevance.

Quantisation

Normalised vector:  [0.6, 0.8],  norm 1.0

int8 symmetric, scale = 0.8 / 127 = 0.0062992:
  q = [round(0.6/0.0062992), round(0.8/0.0062992)]
    = [95, 127]

Dequantised:
  [95 * 0.0062992, 127 * 0.0062992] = [0.598425, 0.800000]
  norm = sqrt(0.358113 + 0.640000) = 0.999056

Close to 1, not 1. Across a corpus the errors are not
symmetric, so the deviations bias comparisons in a
consistent direction rather than cancelling.

If you quantise your vectors, re-normalise after dequantisation, or use a metric that does not care. The effect is small per vector and consistent across millions of them, which is exactly the profile of a bug that shows up as a small unexplained quality regression.

Dimensionality reduction

Projecting a unit vector onto a subspace shortens it, by an amount that depends on how much of that particular vector lived in the discarded directions. After PCA or after truncating a Matryoshka embedding to fewer dimensions, norms vary across the corpus and must be restored.

Norms in training: weight decay and clipping

Norms do a second job that has nothing to do with retrieval. Two of the most common training hyperparameters are norms with a threshold attached, and both are easier to set once you can see the arithmetic.

Weight decay is a multiplicative shrink

L2 regularisation adds (lambda / 2) * ||w||^2 to the loss.
Its gradient is lambda * w, so the update becomes:

  w <- w - lr * (grad + lambda * w)
     = w * (1 - lr * lambda) - lr * grad

The first term shrinks every weight by the same factor
every step, whether or not the data asks it to.

lr = 0.001, lambda = 0.01:
  factor per step = 1 - 0.00001 = 0.99999

Over 100,000 steps with zero gradient:
  0.99999^100,000 = 0.3679 = 1/e

A weight left untouched by the data loses 63% of
its magnitude over a training run. That is the
mechanism: weight decay is a constant pressure
toward zero that only useful weights resist.

The decoupled variant separates the shrink from the adaptive learning rate, so the effective decay does not vary per parameter with the optimiser’s running estimates. It is why the decay coefficient in a decoupled optimiser is not comparable to an L2 coefficient in a plain one, and copying a value between them gives a very different amount of regularisation.

Gradient clipping is a norm with a ceiling

Clipping computes the L2 norm of the whole gradient — every parameter’s gradient concatenated into one vector — and rescales if it exceeds a threshold. The direction is preserved and only the length changes, which is exactly the property you want: one anomalous batch should not move the weights further than a good one.

A useful diagnostic falls out of this. Log the global gradient norm every step. In a healthy run it is fairly stable with occasional spikes; if it is climbing steadily, the model is heading for a divergence some number of steps before the loss shows it, and that is the earliest warning available.

Checking, in one line

import numpy as np

norms = np.linalg.norm(vectors, axis=1)
print(f"min {norms.min():.6f}  max {norms.max():.6f}  "
      f"mean {norms.mean():.6f}  std {norms.std():.6f}")
print("zero vectors:", int((norms < 1e-9).sum()))

# Normalised corpus:
#   min 1.000000  max 1.000000  mean 1.000000  std 0.000000
# Anything else means you are ranking partly on length.

1. Run that on a sample of what is actually in your index, not on what the embedding API returned. The bug is usually introduced between those two points.
2. Run it on your query vectors too. Normalising the corpus and not the queries scales every score by a constant, which does not change the ranking for one query but does make scores incomparable between queries — which breaks any absolute similarity threshold.
3. Check the count of zero vectors explicitly. It is almost never zero in a real corpus, and each one is a latent NaN.
4. If the norms are not 1 and your metric is cosine, nothing is broken — cosine normalises internally. Normalise anyway and switch to inner product, which is faster and gives identical results.

When not to normalise

• When the model was trained with the dot product. Some retrieval models deliberately encode a quality or popularity signal in the norm, and normalising deletes it. The model card will say which metric to use; where it does not, cosine is the safe default for text and worth testing against inner product on your own evaluation set.
• When magnitude is the measurement. Feature vectors of physical quantities, counts, or anything where “twice as much” is meaningful should not be direction-only. Normalisation is a decision about embeddings, not a universal hygiene step.
• When you need to detect degenerate embeddings. An unusually small norm before normalisation is a useful signal that the input was empty, whitespace, or otherwise pathological. Normalise for search, but log the pre-normalisation norm, because afterwards every vector looks equally healthy.
• When you are comparing vectors from two different models. Normalisation does not make them compatible. Different models produce different spaces, and the only migration is to re-embed everything.