Testing That a User Stays in the Same Prompt Variant Across a Session

A user asks a follow-up question and gets an answer in a different format, because the second request landed in the other arm of your prompt experiment. The experiment’s numbers are now noise, and the user thinks the product is broken.

Assignment is a pure function

Every stickiness bug reduces to assignment depending on something other than the key. A random number, a process-local map, a timestamp, a counter, an iteration over the variants. Each of those is invisible in a single-request test and fatal across a session.

So the design rule is that assignment is a hash of a stable identifier and the experiment name, and nothing else. The experiment name in the hash matters: without it, the same user lands in the same bucket position for every experiment, which correlates your experiments with one another and makes two simultaneous tests uninterpretable.

// assign.ts
import { createHash } from "node:crypto";

export function assign(experiment, key, variants, salt = "v1") {
  const digest = createHash("sha256")
    .update(salt + ":" + experiment + ":" + key)
    .digest();
  const bucket = digest.readUInt32BE(0) % 10000;   // 0.01% resolution
  const cutoffs = cumulativeCutoffs(variants);     // e.g. [500, 10000] for 5/95
  return variants[cutoffs.findIndex((c) => bucket < c)];
}

The first two tests write themselves. The same key returns the same variant across a thousand calls, and two different experiment names give the same user independent buckets. Both are exact assertions with no tolerance, which is unusual enough on this ground to be worth taking.

Surviving a restart and a deploy

The strongest version of the determinism test asserts against hard-coded expected values, not merely against consistency within one run.

it("assigns the same variants it assigned yesterday", () => {
  expect(assign("summary-tone", "user-1001", ["a", "b"])).toBe("b");
  expect(assign("summary-tone", "user-1002", ["a", "b"])).toBe("a");
  expect(assign("summary-tone", "user-1003", ["a", "b"])).toBe("a");
});

Those three literals are a golden test for the assignment function itself. Any change to the hash, the salt, the modulus or the cutoff arithmetic re-buckets your entire population, and this test is what turns that from a mystery in the metrics into a failing assertion on the pull request that caused it. Fill the expected values in from a first run; the point is that they are frozen, not that they are predictable in advance.

The corollary is that changing the split is not free. Moving from 5/95 to 10/90 by extending the cutoff keeps every already-assigned user in place and adds new ones, which is what you want. Re-randomising with a new salt moves users across arms mid-experiment, invalidating the comparison. Assert the safe version: for a fixed salt, every key assigned to the variant at 5% is still assigned to it at 10%.

When the key changes underneath you

This is the bug that survives all of the above, because the function is deterministic and the input changed. An anonymous visitor is bucketed on a device identifier, signs in, and is now bucketed on a user id — a different key, an independent draw, a fifty-fifty chance of flipping arms in the middle of the conversation they were having.

The same thing happens whenever the key can be absent. A missing cookie that falls back to a fresh identifier per request is the maximally broken case: assignment is deterministic in the key, and the key is new every time.

1. Assert that a request with no identifier does not silently generate one. It should either use an explicit anonymous key that is stable for the session, or return the control variant. Both are defensible; an unstable key is not.
2. Assert the sign-in case directly: assign with the anonymous key, simulate the identity change, assign again, and require the variant to be unchanged. The usual implementation carries the original assignment forward on the session and only re-derives for a genuinely new session.
3. Assert the key is not the thing you are measuring. Bucketing on a field the experiment can change — a plan tier, a preference — makes the population drift between arms as the experiment runs.

Pinning the variant to the conversation

For an LLM product there is a stricter requirement than per-user stickiness, and it is the one that produces the visible bug in the opening paragraph: a single conversation must be answered by a single prompt version, even if the user’s assignment changes legitimately between turns.

The reason is that the conversation contains its own history. Turn one answered under prompt A is now context for turn two; answering turn two under prompt B gives the model a transcript that contradicts its current instructions, and the output degrades in ways that look like a model problem. Resolve the variant when the conversation is created, store the resolved prompt version with the thread, and read it thereafter.

it("keeps the prompt version pinned for the life of a thread", async () => {
  const thread = await createThread({ userId: "user-1001" });
  expect(thread.promptVersion).toBe("summary@b-3");

  await rolloutStore.set("summary-tone", { b: 0.0, a: 1.0 });   // experiment ended

  const next = await resolvePromptFor(thread);
  expect(next).toBe("summary@b-3");                             // still pinned
});

That test also covers the deploy case, which is the one people are surprised by: an experiment that is concluded and switched off mid-conversation should not change the conversation in flight. Pinning the resolved version rather than the variant name is what makes this work, and it is the same discipline as prompt versioning generally — the identifier you store is the exact artefact, not a pointer that will be re-resolved later. It also has to be the thing you log, or your analysis joins outcomes to whatever the variant is now rather than to what produced them.

Distribution, without a flaky test

One test remains: that the hash spreads keys evenly. Written naively — generate a thousand random keys, assert the share is close to 5% — it fails occasionally and then gets deleted.

Make it deterministic instead. Use a fixed list of keys, not random ones, and a wide tolerance derived from the count rather than from taste. With 100,000 fixed keys at a 5% split the expected count is 5,000, and the standard deviation is the square root of 100000 x 0.05 x 0.95, about 69 — so a tolerance of 400, nearly six standard deviations, will never fail by chance and still catches every real bug, because real bugs here are not subtle. A modulus of 100 where you meant 10,000, a cutoff comparison with the wrong operator, or a hash truncated to one byte all miss by enormous margins. The arithmetic behind that expected count is worked out in the rollout reach calculation.