Interaction Effects When Two Prompt Experiments Run at Once

One team is testing a rewritten system prompt. Another is testing a warmer tone instruction. Both are live, both are 50/50, and nobody coordinated. Most of what people fear about this situation is not true, and the two things that are true are not statistical.

The design you are running whether you meant to or not

Two independent binary experiments produce four cells. If each is randomised independently at 50/50, traffic distributes evenly across them, and what you have is a 2 × 2 factorial design — a well-studied object that experimental statistics has been comfortable with since the 1920s.

                     tone: control    tone: warm
system: control          25%             25%
system: rewritten        25%             25%

The vocabulary is worth having. The main effect of the system prompt is the average difference it makes, collapsing across the tone arms. The interaction is whether the system prompt’s effect depends on which tone arm the request is in — whether the two changes are more, or less, than the sum of their parts.

The main effects are fine

This is the part that surprises people, and it is worth being exact about because a lot of unnecessary sequencing of experiments follows from getting it wrong.

Because tone assignment is independent of system-prompt assignment, both arms of the system-prompt experiment contain the same mix of tone conditions. The tone experiment therefore adds variance to the system-prompt comparison, but it does not add bias — it is spread evenly across the thing you are measuring, exactly like every other source of user heterogeneity you never controlled for. The estimate of the system-prompt main effect is unbiased whether or not you know the tone experiment exists.

So the common instinct — freeze all other experiments while mine runs — buys very little and costs a great deal of calendar time. What it does buy is a slightly tighter interval, since the tone effect is one less source of residual variance, and interpretability of the result as “the effect in the world as it was” rather than “averaged over the tone conditions”. Neither is usually worth a month of queueing.

There is a real multiplicity cost, though, and it is separate. Run m independent experiments each at α = 0.05 and the probability that at least one produces a false positive is:

P(at least one false positive) = 1 - 0.95^m

m =  1 :  0.050
m =  5 :  0.226
m = 10 :  0.401
m = 20 :  0.642

Twenty concurrent experiments and it is more likely than not that one of them is a lie. That is an organisational fact rather than an interaction effect, and the usual answer is to hold experiments to a pre-registered primary metric and treat everything else as exploratory.

The interaction costs four times the sample

The thing you genuinely cannot measure without planning for it is the interaction. Write the four cell means as y11, y10, y01 and y00, each estimated from n observations with per-observation variance σ².

main effect of system prompt
    = (y11 + y10)/2  -  (y01 + y00)/2
    Var = (1/4) * 4 * (sigma^2 / n)  =  sigma^2 / n
    SE  = sigma / sqrt(n)

interaction
    = (y11 - y10)  -  (y01 - y00)
    Var = 4 * (sigma^2 / n)
    SE  = 2 * sigma / sqrt(n)

The interaction's standard error is twice the main effect's,
so detecting an interaction of the same magnitude with the same
power needs 4x the observations per cell.

Four times. And interactions are typically smaller than main effects, not the same size, so the realistic requirement is worse than fourfold. An experiment powered to detect a two-point change in the main effect is powered to detect essentially nothing about whether the two changes conflict.

The practical consequence is not “always power for the interaction”, which is unaffordable. It is that you should stop reading the interaction estimate as evidence. A non-significant interaction from a study powered for main effects is not a finding that the changes are additive; it is a study that could not have told you either way. Say “we did not measure this” rather than “there was no interaction”.

The failure that does bias it: a shared hash

All of the above assumes independent randomisation. Here is how that assumption dies in a real codebase.

Assignment is usually implemented as a hash of a stable identifier, bucketed. Somebody writes a helper, and the helper hashes the user id. Both experiments call the helper. Now bucket 37 is bucket 37 in both experiments, so every user in the system-prompt treatment is also in the tone treatment, and the design is not 2 × 2 at all:

                     tone: control    tone: warm
system: control          50%              0%
system: rewritten         0%             50%

Two of the four cells are empty. Every effect you attribute to the system prompt is the combined effect of both changes, and there is no data anywhere that can separate them. This is not a subtle loss of power; it is total confounding, and it is invisible unless you look at the cell counts.

The fix is one line and it is to salt the hash per experiment, so that each experiment’s bucketing is independent of every other:

// Wrong: the same bucket for every experiment.
const bucket = hash(userId) % 100;

// Right: the experiment key is part of the hashed input, so
// bucketing is independent across experiments for the same user.
const bucket = hash(experimentKey + ":" + userId) % 100;

Then assert the cell counts. A single check that all four cells hold roughly a quarter of traffic catches this, catches an accidental filter on one arm, and catches a rollout that silently stopped assigning. Emit the counts as a metric on the canary panel described in what to compare between canary and baseline.

The LLM-specific failure: one prompt, two edits

This one has no analogue in UI experimentation and it is the reason the question is worth its own page.

Two UI experiments change two different parts of a page. Two prompt experiments frequently change the same string. If the system prompt is assembled from fragments — a role section, a policy section, a tone section, a formatting section — then experiment A rewriting the role section and experiment B substituting the tone section produce, in cell (1,1), a system prompt that no one has ever read. It is not a combination of two reviewed prompts; it is a fourth prompt that came into existence at request time.

The failure modes are concrete. The rewritten role section may already contain a tone instruction, so cell (1,1) contains two contradictory tone directives and the model follows whichever is later or more emphatic. The new section may push the total past a length where the model reliably attends to the formatting rules at the end. Or the two fragments may simply not be grammatical when concatenated, which matters more than it sounds because a malformed instruction is frequently ignored rather than partially followed.

The defence is to make the four prompts artefacts rather than accidents. Render every cell at build time, hash each rendered prompt, and assert in CI that the set of hashes matches a checked-in list of reviewed prompts:

import { describe, expect, it } from "vitest";
import { renderSystemPrompt } from "../src/prompt";
import { createHash } from "node:crypto";

const cells = [
  { system: "control", tone: "control" },
  { system: "control", tone: "warm" },
  { system: "rewritten", tone: "control" },
  { system: "rewritten", tone: "warm" },
];

describe("experiment cells", () => {
  it("renders only reviewed prompt combinations", () => {
    const hashes = cells.map((c) =>
      createHash("sha256").update(renderSystemPrompt(c)).digest("hex").slice(0, 12),
    );
    expect(hashes.sort()).toEqual(REVIEWED_PROMPT_HASHES.sort());
  });
});

The assertion is on the hash set, not on the text, which is what makes it maintainable: the test fails whenever a combination appears that nobody signed off, and the fix is to read that combination and add its hash. It is the same discipline as keeping prompts in version control in the first place, covered in storing prompts as files.

What to do

• Do not serialise experiments by default. Under independent randomisation the main effects are unbiased, and queueing costs more than it saves.
• Salt every assignment hash with the experiment key, and assert the cell counts rather than trusting that you did.
• Serialise the specific pairs that touch the same prompt region. Not because of statistics — because cell (1,1) is an unreviewed prompt. If they must overlap, enumerate and review all four rendered prompts.
• Never report an interaction from a main-effect-powered study. The estimate exists; it means nothing at four times the standard error.
• Count the concurrent experiments and know your family-wise error rate. At ten, two in five of your “wins” being spurious is the baseline expectation, not a worst case.