Testing That an Embedding Model Returns Stable Vectors for the Same Input

Two calls to an embedding endpoint with byte-identical input can return different vectors. Not wildly different — different in the last few decimal places — which is enough to break an equality assertion, a cache keyed on the vector, or a deduplication rule that assumed determinism.

The same text does not give the same vector

This surprises people because embeddings feel like a pure function of the text. Nothing is being sampled, there is no temperature, and the model has no state — so the natural assumption is that the output is deterministic. In practice, repeated calls to a hosted embedding endpoint with the same input can return vectors that differ slightly, and developers have been reporting exactly this against hosted models since 2023.

What follows from it depends entirely on what you built. If you store the vector and never re-embed, nothing follows. If you compare a freshly embedded query against a stored vector, nothing follows either, because the difference is far below any threshold you would use. But if you hash a vector to use as a cache key, deduplicate by exact vector equality, or write a test asserting toEqual on an embedding, all three break intermittently and in a way that looks like a bug in your code.

Why, mechanically

Floating-point addition is not associative: for finite-precision values, (a + b) + c and a + (b + c) can give different results. A forward pass through a model is an enormous number of accumulations, and the order in which those accumulations happen is not fixed. It depends on which GPU kernel was chosen, which depends on the shape of the batch, which depends on how many other requests arrived at the same moment.

So the same text embedded alone and embedded in the middle of a batch of two hundred can take numerically different paths to the same mathematical answer, and the results differ in the low-order bits. This is not a defect in the provider and it is not fixable by a seed — the same mechanism is why fixing a seed does not make a language model bit-identical either. It is a property of running a large reduction across parallel hardware.

Choosing a threshold instead of asserting equality

The assertion that survives is a similarity floor. Cosine similarity between two runs of the same input should be extremely close to 1 — far closer than any two genuinely different texts would be — and the test is that it stays there.

A useful simplification: OpenAI’s embeddings guide states that its embeddings are normalised to length 1, which means cosine similarity can be computed slightly faster as a plain dot product. Do not assume this of every provider or of a model you host yourself, and note that the same documentation warns that if you shorten an embedding using the dimensions parameter you must normalise it again afterwards — a truncated vector is no longer unit length, and a dot product on it is no longer cosine similarity. Write the general cosine function anyway; it is four lines and it removes an assumption from your test.

Pick the floor from the data rather than from taste. Embed a handful of representative strings ten times each, record the minimum pairwise similarity you observe, and set the threshold a little below it. Then say in a comment what the observed value was and when — a threshold with no provenance is one that gets loosened the first time it fails.

The test

// embedding-stability.test.ts
import { describe, expect, it } from "vitest";
import { client } from "./probe";

const MODEL = "text-embedding-3-small";
const STABILITY_FLOOR = 0.9999;

function cosine(a: number[], b: number[]): number {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    na += a[i] * a[i];
    nb += b[i] * b[i];
  }
  return dot / (Math.sqrt(na) * Math.sqrt(nb));
}

async function embed(input: string | string[]) {
  const res = await client.embeddings.create({ model: MODEL, input });
  return res.data.map((d) => d.embedding);
}

describe("embedding stability", () => {
  const text = "Refund requested for order 88031, item damaged in transit.";

  it("repeated calls stay above the stability floor", async () => {
    const runs = [];
    for (let i = 0; i < 5; i++) runs.push((await embed(text))[0]);

    for (let i = 1; i < runs.length; i++) {
      const sim = cosine(runs[0], runs[i]);
      expect(sim, "run " + i + " similarity " + sim).toBeGreaterThan(STABILITY_FLOOR);
    }
  }, 60_000);

  it("returns the documented dimensionality", async () => {
    const [vec] = await embed(text);
    expect(vec).toHaveLength(1536);
  });

  it("distinguishes this text from an unrelated one", async () => {
    const [a] = await embed(text);
    const [b] = await embed("The 14:02 train to Utrecht departs from platform 3.");
    expect(cosine(a, b)).toBeLessThan(0.5);
  });
});

The third test is not padding. A stability test alone passes trivially if the endpoint is broken and returns the same constant vector for every input — a failure mode that has happened, usually via a misconfigured local server or a cache with too coarse a key. One negative case makes the suite meaningful, and picking a genuinely unrelated sentence keeps the bound loose enough not to be flaky.

Batch position is the other variable

The test above sends one input at a time, which is the easy case. The interesting one is whether a text embedded alone matches the same text embedded as element 47 of a batch of 100 — because that is exactly the difference between how you index a corpus and how you embed a query, and batching is where the kernel shapes change.

it("is stable across batch position", async () => {
  const target = "Refund requested for order 88031.";
  const filler = Array.from({ length: 99 }, (_, i) => "Unrelated record " + i);

  const [alone] = await embed(target);
  const batch = await embed([...filler.slice(0, 47), target, ...filler.slice(47)]);

  expect(cosine(alone, batch[47])).toBeGreaterThan(STABILITY_FLOOR);
}, 60_000);

If that one fails while the single-input test passes, you have found something worth knowing before you index a million documents: index and query embeddings are not directly comparable at the precision you assumed, and any threshold you tune on one will be slightly wrong on the other. It is also the test to run after changing your batch size, which is otherwise an invisible change to the numbers in your index.

Two consequences of all this are worth acting on beyond the tests. First, do not key a cache on a vector. Key it on a hash of the input text and the model id, which is stable by construction; a vector used as a key produces a cache that never hits and grows without bound. Second, if you deduplicate documents by embedding similarity rather than by content hash, the threshold you use for that is subject to exactly the drift described on the threshold page, and a deduplication rule that quietly loosens is a corpus that quietly fills with near-duplicates.

Where should this suite run? Not on every commit — it makes real requests and the thing it watches for changes on the order of a model release. Nightly is right, plus on any change to the embedding model, the dimension setting or the batching code. And record the numbers it observes rather than only its pass or fail: a similarity that has drifted from 0.99999 to 0.9997 without crossing your floor is still news, and it is the kind of news that arrives before the failure does.