Prompt Sensitivity: Same Question, Different Answer

Change a colon to a hyphen in your prompt template. Accuracy moves. Nothing about the meaning changed, nobody would call the two prompts different, and yet the number on your evaluation dashboard is now a different number. This is one of the most under-reported facts about working with these systems.

The finding

Sclar, Choi, Tsvetkov and Suhr’s Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design (2023), the FormatSpread paper, is the careful version. They defined a space of formatting choices that leave meaning untouched — the separator between a field name and its value, the space after it, casing, the enumeration style, the wrapper around the answer — and searched it. Across tasks and models they reported accuracy spreads that reached tens of points, with a headline case of up to 76 accuracy points between two formats on LLaMA-2-13B. Models that appeared ranked differently under one format changed places under another.

Mizrahi et al.’s State of What Art? (TACL, 2024) made the evaluation-methodology argument that follows: single-prompt benchmark scores are not reproducible claims about a model, and multi-prompt evaluation — reporting a distribution over semantically-equivalent instructions — changes model rankings. Lu et al. (2022) had shown the same for the order of few-shot examples: permuting the examples alone spans a wide accuracy range, and the good orderings do not transfer between models.

None of this is about temperature. These are argmax decodes on different strings; the variance is in the model’s response surface, not in the sampler. That distinction matters, because the usual reflex — set temperature to 0 — does nothing about it. For the separate question of why temperature 0 is not even deterministic, see non-determinism at temperature zero.

Which surfaces move the answer

In rough order of how much they tend to matter, and all of them are things a reasonable engineer changes without thinking:

• Field separators and whitespace. Question: X versus Question - X versus QUESTION:: X. Also the number of newlines between sections.
• Option labels and their order. Multiple-choice answers are labelled A/B/C/D or 1/2/3/4, and models show measurable position bias — Zheng et al. reported systematic preferences for particular option positions independent of content, which means a benchmark that never permutes options is measuring the bias as much as the capability.
• Where the instruction sits. Before the data or after it. For long contexts this interacts with position effects and the difference can be large.
• Chat template details. Whether your content lands in the system or the user role, and whether the template the provider applies matches the one the model was tuned with.
• Paraphrase of the instruction itself. The most obvious axis and, interestingly, often not the largest one.

Why a separator changes an answer

Nothing here is a bug, and understanding why makes it predictable.

Tokenisation comes first. “: ” and “:” are different tokens, and a change in one token changes every subsequent hidden state. The model does not see a semantic gloss of your prompt; it sees a specific sequence, and the function it computes is not smooth over edits that a human considers equivalent.

Then training distribution. Formats that appeared frequently in pretraining and in instruction tuning sit in a well-populated region of the input space where the model’s behaviour is well determined by data. An unusual format sits in a sparse region where behaviour is determined by whatever the network generalised, which is much more variable. This is why the formats that win are usually the boring ones — Markdown headings, JSON, the vendor’s own documented template.

And the accumulation: the effect compounds along the sequence, so a perturbation early in a long prompt has more room to change the trajectory than the same perturbation near the end.

The harness

The point is not to find the best prompt. It is to know how much your number moves, so that you know whether a change you are about to ship is a real improvement or a resample of the same distribution.

import statistics, itertools

SEPARATORS = [": ", " - ", " = ", ":\n"]
CASINGS    = [str, str.upper, str.title]
ORDERS     = ["instruction_first", "data_first"]

def variants(instruction, data):
    for sep, case, order in itertools.product(SEPARATORS, CASINGS, ORDERS):
        head = case("instruction") + sep + instruction
        body = case("input") + sep + data
        yield head + "\n\n" + body if order == "instruction_first" \
              else body + "\n\n" + head

def spread(eval_set):
    """One accuracy per format. Report the distribution, never the max --
    picking the best format on your eval set is overfitting to it."""
    accs = []
    for prompt_fn in range(len(SEPARATORS) * len(CASINGS) * len(ORDERS)):
        correct = 0
        for item in eval_set:
            v = list(variants(item["instruction"], item["data"]))[prompt_fn]
            correct += is_correct(call_model(v, temperature=0), item["gold"])
        accs.append(correct / len(eval_set))
    return {"mean": statistics.mean(accs), "sd": statistics.pstdev(accs),
            "min": min(accs), "max": max(accs), "spread": max(accs) - min(accs),
            "n_formats": len(accs)}

Twenty-four formats times a two-hundred-item evaluation set is 4,800 calls, which for a small model is minutes and pennies. Run it once per model you are considering. The output you care about is spread: it is the resolution limit of every single-prompt comparison you will make afterwards.

One refinement is worth the extra loop. Record per-item results rather than only per-format accuracies, and you get two things for free: the items that every format gets right (which are wasted evaluation budget) and the items whose answer changes with the format (which are the genuinely fragile ones). The second set is small, and reading twenty of them by hand explains more about a model’s behaviour on your task than the aggregate ever will.

What to do with the spread

• Treat spread as your error bar. If reformatting moves accuracy by six points, a four-point improvement from a cleverer prompt is not an improvement. Most reported prompt-technique wins are inside somebody’s unmeasured spread.
• Do not tune the format on the evaluation set. The best-performing format on 200 items is partly noise, and it will not hold. Pick a conventional format, hold it fixed, and spend the effort on the content of the prompt.
• Re-run it on a model change. Format preferences do not transfer between families, and a prompt tuned against one model is carrying that model’s idiosyncrasies into the next.
• Use disagreement as a signal. Where several formats of the same question produce different answers, that item is one your system is uncertain about — a cheap, per-item uncertainty measure that needs no logprobs and follows the same logic as the sampling methods on the calibration page.