Cosine Similarity vs Dot Product vs Euclidean

This question generates more forum argument than it deserves, because for the majority of real setups the answer is provably “it makes no difference at all”. Here is the proof, and then the minority of cases where it makes a large one.

The three definitions

dot(a, b)       = sum over i of a[i] * b[i]
cosine(a, b)    = dot(a, b) / (|a| * |b|)
euclidean(a, b) = sqrt(sum over i of (a[i] - b[i])^2)

where |a| = sqrt(dot(a, a))

Dot product is unnormalised and grows with the magnitude of either vector. Cosine divides that magnitude out and measures only the angle, giving a value in [-1, 1]. Euclidean is straight-line distance, so smaller is better where the other two have larger as better.

That last difference is a real source of bugs even where the metric choice is irrelevant. Half the vector libraries return a similarity and half return a distance, some return “cosine distance” defined as 1 - cosine, and a filter written as score > 0.8 against a distance keeps exactly the results it was meant to exclude. Before you tune any threshold, print the score for a document against itself. If it is 1.0 you have a similarity; if it is 0.0 you have a distance.

On unit vectors they are the same ranking

Suppose every stored vector has been normalised to unit length, so |a| = |b| = 1. Most modern embedding APIs return unit vectors already; check once with a quick norm computation rather than assuming. Under that assumption:

cosine(a, b) = dot(a, b) / (1 * 1) = dot(a, b)                    (1)

euclidean(a, b)^2 = |a|^2 + |b|^2 - 2 * dot(a, b)
                  = 1 + 1 - 2 * dot(a, b)
                  = 2 - 2 * dot(a, b)                              (2)

so  euclidean(a, b) = sqrt(2 - 2 * cosine(a, b))

Line (1) says cosine and dot product are the same number, not merely the same ordering. Line (2) says squared Euclidean distance is a strictly decreasing linear function of cosine similarity — and a strictly monotone transform cannot reorder anything. Sorting ascending by Euclidean distance and descending by cosine produces the identical list, in the identical order, for every query.

So if your vectors are normalised, the metric debate is settled before it starts. Your top-10 is byte-identical whichever of the three you configure. What differs is the number printed next to each result: cosine 0.94 corresponds to Euclidean sqrt(2 - 1.88) = 0.346, and any threshold you have tuned must be translated through that relation rather than carried over.

When the choice genuinely matters

Vectors that are not normalised

This is the entire real disagreement. If magnitudes vary, dot product rewards long vectors and cosine ignores length. Whether that is a feature depends on what length encodes in your model. Some retrieval models are trained so that norm carries a notion of confidence or informativeness, and their authors specify inner product as the scoring function; using cosine there discards a signal the model was trained to emit. Conversely, if length is an artefact — longer chunk, more tokens, bigger norm — dot product turns your search into a popularity contest won by your longest documents.

The rule is not “cosine is safer”. The rule is: use the function the model card names, because the model was trained against that function.

Non-embedding vectors in the same table

Vector columns get reused for things that are not embeddings — counts, TF-IDF rows, feature vectors, user-behaviour tallies. None of those are normalised and none of the reasoning above applies. There, magnitude is usually real information and Euclidean or dot product is often correct.

Documents of very different length in one index

Even with a model that returns unit vectors, mixing 30-token titles with 2,000-token pages puts two different length regimes in one space, and pooling behaves differently across them. Cosine does not fix this — the vectors are already normalised, so there is no magnitude left to divide out — which is worth stating because “use cosine” is the usual advice offered for it. The fix is chunking to a consistent size, not a metric change.

After quantisation

Binary quantisation replaces the metric entirely: a 1-bit vector is compared with Hamming distance, which is a popcount over an XOR and bears no arithmetic relation to the cosine of the originals. Scalar int8 quantisation preserves ordering approximately but not exactly, so a similarity threshold calibrated on float32 will not hold. Any threshold you set must be re-derived after a quantisation change.

The mismatch that disables your index

Here is the one that costs people a weekend. In pgvector, each metric has its own operator and its own index operator class, and an index built for one metric cannot serve a query written with another:

<->   L2 distance          vector_l2_ops
<#>   negative inner product vector_ip_ops
<=>   cosine distance       vector_cosine_ops
<+>   L1 distance           vector_l1_ops

CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops);

SELECT id FROM items ORDER BY embedding <-> $1 LIMIT 10;   -- seq scan!
SELECT id FROM items ORDER BY embedding <=> $1 LIMIT 10;    -- index scan

The first query does not error. It returns correct results, computed by reading every row in the table, and it will do that in production forever while somebody wonders why vector search is slow at 400,000 rows. EXPLAIN is the whole diagnosis: if you see a Seq Scan rather than an Index Scan on your vector column, the operator and the opclass disagree.

The <#> operator returning negative inner product is not a quirk to work around either. Postgres index scans return rows in ascending order, so a similarity that should be maximised has to be negated to be minimised. Your ORDER BY is correct as written; just remember to flip the sign before showing a score to anyone.

How to decide in practice

• Read the model card. If it names a scoring function, that is the answer and this page is over.
• If it does not, check whether the vectors are unit length. Compute the norm of a few. If they are all 1.000, pick any metric — you are in the proof above — and pick cosine so that scores are readable.
• If norms vary and you are unsure whether that is signal, normalise. Discarding a signal you do not understand is a smaller error than amplifying an artefact you have not noticed.
• Normalise before insert, not at query time, and store the normalised vector. Then dot product is the cheapest correct choice and your index never has to compute a square root.