What a Migration Does to an Existing Prompt's Persona Consistency

The system prompt did not change. The persona document did not change. Support is nonetheless forwarding messages that say the assistant “sounds different” — more formal, or chattier, or suddenly fond of bulleted lists. This is a real regression with a specific cause, and it is fixable in the prompt.

The report you will get

The complaint arrives as an impression rather than a defect, which is why it tends to sit unresolved. “It doesn’t feel like us any more.” Push for specifics and you will usually get some subset of: responses got longer; it started opening with a restatement of the question; it uses headings where it used to write a paragraph; it stopped using contractions; it added emoji, or dropped them; it hedges more; it ends every answer by offering to do something else.

Every item on that list is a surface feature of the text, and every one of them is countable. That is the route out of an unfalsifiable discussion, and the rest of this page is the mechanism and the fix. The general question of whether a prompt carries across models is covered in prompt portability; what is specific to persona is that the drift is invisible to every correctness test you already have. The answers are still right.

The persona had two authors

Read your system prompt as if you had never seen the assistant it produces. It probably says something like “You are Ada, a friendly and knowledgeable support assistant for Acme. Be helpful and concise.” Now ask which of the observed behaviours that text determines. Whether to use headings? Not stated. How long an answer should be? “Concise” — relative to what? Whether to open with an acknowledgement? Not stated. Whether to use contractions? Not stated.

The persona your users recognised was the union of what you specified and what the model chose in the gaps. The gaps are large, and the model’s choices in them are stable enough over a long deployment that they read as intentional. Migrate and the second author is replaced. The parts you wrote carry over exactly; the parts you never wrote are rewritten wholesale.

Three of the model’s defaults account for most complaints. Length calibration: how long an answer should be for a given question, which differs sharply between models and is why “concise” means different things on either side of a swap. Structural default: whether the natural reply to a multi-part question is prose or a formatted list. Instruction literalism: a model that follows the letter of an instruction applies “be friendly” more uniformly than one that reads it as a general disposition, which is why the same word can produce warmth in one model and relentless cheerfulness in another.

Making the drift countable

Take a fixed set of prompts — a hundred real user turns from production logs is ideal — and run them against both models with the identical system prompt. Then compute the same features over both output sets. None of this requires a judge model.

FEATURES = {
    "chars":          lambda t: len(t),
    "sentences":      lambda t: len(re.findall(r"[.!?](?:\s|$)", t)),
    "contractions":   lambda t: len(re.findall(r"\b\w+'(?:s|re|ve|ll|t|d)\b", t, re.I)),
    "bullets":        lambda t: len(re.findall(r"^\s*[-*•]\s", t, re.M)),
    "headings":       lambda t: len(re.findall(r"^#{1,6}\s", t, re.M)),
    "emoji":          lambda t: len(EMOJI_RE.findall(t)),
    "second_person":  lambda t: len(re.findall(r"\byou(?:r|rs)?\b", t, re.I)),
    "hedges":         lambda t: len(re.findall(
                          r"\b(?:might|perhaps|generally|typically|it depends)\b", t, re.I)),
    "offer_ending":   lambda t: int(bool(re.search(
                          r"(?:let me know|would you like|anything else)[^.!?]*[.!?]\s*$", t, re.I))),
}

def profile(outputs):
    return {k: median(f(t) for t in outputs) for k, f in FEATURES.items()}

before, after = profile(old_outputs), profile(new_outputs)
for k in FEATURES:
    if before[k] and abs(after[k] - before[k]) / before[k] > 0.25:
        print(f"{k}: {before[k]} -> {after[k]}")

The output is a short list of the two or three features that actually moved, which converts “it sounds different” into “median response length rose 60% and bullet count tripled”. That is a fixable statement. It also tells you what not to touch: features that did not move need no prompt instruction, and adding one for them only makes the prompt longer and more brittle.

Specifying what was implicit

Now write the gaps down. Three techniques, in increasing order of strength.

Replace relative words with observable rules. “Concise” is a comparison with no referent. “Answer in at most three sentences unless the user asks for detail” is checkable by the model and by your test. Do this for every feature that moved, and only those.

State the structural default explicitly. If the old assistant wrote prose, say so: “Reply in continuous prose. Use a bulleted list only when enumerating three or more discrete items the user must act on separately; never use headings.” A model does not infer “we are a prose shop” from a persona adjective.

Show, do not only tell. Two or three short exemplar exchanges in the persona are the strongest available signal, because the model matches their register, length and structure directly rather than interpreting an adjective. Choose exemplars that span the range — a one-line factual answer, a refusal, a multi-step explanation — because a single exemplar pins one shape and the model extrapolates the rest. Label them as illustrative so they are not read as content.

A persona regression test

Persona is not a one-time fix, because the next migration will do the same thing. Turn the measurement into a test that runs in CI.

1. Freeze a persona fixture: forty to sixty user turns covering your real distribution, including at least one refusal case and one case where the user is annoyed. Persona breaks first under pressure, and a fixture of happy-path questions will not catch it.
2. Record the feature profile of the current, approved deployment as a JSON baseline checked into the repository next to the prompt.
3. On every prompt change or model change, regenerate and assert each feature stays within a tolerance band. Derive the band from run-to-run variance on the same model, not from a guess — sampling noise gives you a floor below which a threshold is just a flaky test.
4. On failure, print the feature deltas and three example outputs side by side with their baseline counterparts. A reviewer can then judge in seconds whether the drift is acceptable, and approve it by committing the new baseline.

The last step is what makes this survivable. A persona test that can only fail becomes noise and gets disabled. A persona test that fails loudly and is approved by updating a committed baseline turns every persona change into a reviewed diff — which is what you wanted from the beginning, and what you did not have when the migration silently rewrote half of it.