Vectors, Dot Products, and Why Similarity Is an Angle

A dot product is two numbers in disguise: how long the vectors are, and the angle between them. Everything about embedding similarity follows from separating those two, and it can be done on paper with two-dimensional vectors before it is done in a database with 1,536.

A vector is a list with a direction

A vector is an ordered list of numbers. In two dimensions you can draw it as an arrow from the origin: a = [3, 4] is three across and four up. Its length — written ||a||, called the norm — is Pythagoras:

||a|| = sqrt(3^2 + 4^2) = sqrt(9 + 16) = sqrt(25) = 5

That formula is unchanged in any number of dimensions: square every component, sum, square-root. A 1,536-dimensional embedding has a length computed exactly this way, over 1,536 terms instead of two.

The dot product, computed twice

The dot product multiplies matching components and adds the results. Take a = [3, 4] and b = [4, 3]:

a . b = (3 * 4) + (4 * 3) = 12 + 12 = 24

One number out of two lists. There is a second, entirely equivalent definition, and holding both in mind is the whole trick:

a . b = ||a|| * ||b|| * cos(theta)

with ||a|| = 5, ||b|| = sqrt(16 + 9) = 5:

  24 = 5 * 5 * cos(theta)
  cos(theta) = 24 / 25 = 0.96

So a dot product of 24 between two vectors of length 5 means an angle whose cosine is 0.96. The dot product is big when the vectors are long, and big when they point the same way, and by itself it cannot tell you which of those it was.

Turning the dot product into an angle

Divide out the lengths and only the angle is left. That is cosine similarity, and it is one line:

cos_sim(a, b) = (a . b) / (||a|| * ||b||)
              = 24 / (5 * 5)
              = 0.96

theta = arccos(0.96) = 0.2838 radians = 16.26 degrees

Sixteen degrees apart, on a scale where 0 degrees is identical direction and 90 degrees is unrelated. Check the extremes with two more vectors:

c = [-4, 3]:
  a . c = (3 * -4) + (4 * 3) = -12 + 12 = 0
  cos_sim = 0 / (5 * 5) = 0        -> 90 degrees, orthogonal

d = [6, 8]  (which is exactly 2a):
  a . d = 18 + 32 = 50
  ||d|| = sqrt(36 + 64) = 10
  cos_sim = 50 / (5 * 10) = 1.0    -> 0 degrees, same direction

The third case is the one that matters in practice. d is twice a and points in exactly the same direction, so cosine similarity calls them identical while the raw dot product calls d twice as relevant. Which of those you want decides an entire retrieval design, and getting it wrong produces a specific, silent bug.

Nothing changes at 1,536 dimensions

The formulas above have no 2 in them anywhere. They sum over however many components there are. Here is the whole of embedding similarity, in the same six lines that handled [3, 4]:

import numpy as np

def cos_sim(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

a = np.array([3.0, 4.0])
b = np.array([4.0, 3.0])
print(cos_sim(a, b))          # 0.96

# identical function, 1536 dimensions
q = np.random.randn(1536)
d = np.random.randn(1536)
print(cos_sim(q, d))          # near 0

That last line is worth dwelling on. Two random 1,536-dimensional vectors are almost exactly orthogonal, and the higher the dimension the more reliably so. In two dimensions, two random arrows are 90 degrees apart on average but frequently much closer. In 1,536, essentially everything is perpendicular to essentially everything else, which is why an embedding space has room for millions of distinguishable meanings, and also why a cosine similarity of 0.3 can be a strong signal in a space where the baseline is 0.0.

The practical consequence: a similarity score is only interpretable against the distribution of scores from the same model. A threshold of 0.8 copied from one embedding model to another means nothing, and the models genuinely differ in how they spread their scores.

Adding and subtracting vectors

Addition and subtraction work componentwise, and geometrically they compose displacements: a + b is the arrow you get by walking along a and then along b.

a = [3, 4]   b = [4, 3]

a + b = [7, 7]        ||a + b|| = sqrt(98)  = 9.899
a - b = [-1, 1]       ||a - b|| = sqrt(2)   = 1.414

Note that ||a|| + ||b|| = 10, and ||a + b|| = 9.899.
The sum of the lengths is never smaller than the length
of the sum. That is the triangle inequality, and it holds
in every dimension.

Subtraction is the operation behind the analogy arithmetic that made word embeddings famous: king - man + woman lands near queen. The mechanism is that king - man is a displacement vector encoding roughly “royalty, minus maleness”, and adding it to woman applies the same displacement elsewhere in the space.

It is worth being precise about how well that works, because it is usually oversold. The result vector is not queen; it is a point near queen, and the nearest-neighbour search that finds queen normally has to exclude the three input words explicitly, because the closest vector to king - man + woman is very often king itself. The analogies that work reliably are the ones with a consistent, high-frequency relationship — capital cities, plurals, verb tenses. Analogies over anything subtler fail often enough that the technique is a demonstration rather than a tool.

Subtraction is also how you remove an unwanted direction. If you can identify a direction encoding something you do not want to retrieve on — document length, formatting style, a dominant topic — projecting it out is one subtraction per vector:

Remove the component of v along unit vector u:

  v_clean = v - (v . u) * u

Check it worked:
  v_clean . u = (v . u) - (v . u) * (u . u)
              = (v . u) - (v . u) * 1
              = 0

The cleaned vector is exactly orthogonal to u.

The same two lines remove the mean from a corpus of embeddings, which is the first step of PCA and a cheap thing to try when every pair of vectors in your index has a suspiciously high similarity: a large shared mean component inflates every score without carrying any information that distinguishes documents.

Projection, and what a query really does

The dot product also answers “how much of a lies along b?” That is the scalar projection:

proj_length = (a . b) / ||b||
            = 24 / 5
            = 4.8

The vector projection points along b with that length:
  proj = (4.8 / 5) * b = 0.96 * [4, 3] = [3.84, 2.88]

This is exactly what happens inside attention. The scores are Q @ K.T: every query vector dotted with every key vector, asking how much each key lies along each query. Then those scores go through softmax to become weights, and the weights mix the value vectors. Three of the four steps in attention are the arithmetic on this page.

Where the geometric intuition misleads

• “Similar meaning” is not the same as “small angle”. The angle is small when the model was trained to make it small. Antonyms often sit very close together because they appear in identical contexts — hot and cold are distributionally near-twins. A retrieval system that treats cosine similarity as semantic agreement will happily return the opposite of what was asked.
• Cosine similarity has no absolute meaning. It is not a probability and it is not a percentage. A score of 0.72 is informative only in comparison to other scores from the same model on the same kind of text.
• Averaging vectors is not averaging meanings. The mean of two embeddings is a real vector in the space, but the space is not linear in meaning, so the midpoint of “dog” and “aeroplane” does not name anything. Averaging works well enough for pooling many similar things and badly for combining two unrelated ones.
• Zero vectors break everything. Cosine similarity divides by the norm, and an empty-string embedding, or a vector zeroed by a bad conversion, gives a division by zero and then a NaN that propagates through the whole result set.