Testing That Paraphrased Inputs Get Consistent Answers

Same meaning, different wording, same answer. It is the most obvious property a language model should have and one of the easiest to violate. Turning it into a test that fails only when something is actually wrong takes two decisions, and most implementations skip the second one.

Decide what consistent means first

“Consistent” is not one thing, and picking the wrong definition is what makes these tests flaky. Choose by the shape of your output, and always reach for the strictest form that applies:

• Classification. Identical label. Exact string equality. Nothing weaker is needed and nothing weaker should be used.
• Extraction. Identical field values for the fields the paraphrase did not touch. Also exact equality, on the parsed object.
• A number. Within a stated tolerance. Say why the tolerance is what it is — rounding, currency minor units, a genuine range of defensible answers — because an unexplained tolerance grows every time the test fails.
• A ranking. Same top result, or a rank correlation above a bound. Top-1 stability is usually the thing anyone actually depends on.
• Free prose with no extractable decision. Semantic similarity above a threshold, which is the weakest option on this list and needs a defensible threshold behind it.

The rule that follows: never compare two prose answers when you could have extracted a decision and compared that. If the prompt does not currently emit a structured decision field, adding one is a smaller change than building a similarity harness, and it makes every other test in the suite better too.

The paraphrase set, and where it comes from

Two sources, and they play different roles. Mechanical variants are generated in the test: whitespace changes, reordering independent list items, casing of a field the answer must not depend on, swapping a name from a fixed table. These are provably meaning-preserving, so a failure is unambiguous, and they are the right job for a property generator.

Genuine paraphrases — different sentence structure, different vocabulary, a question asked as a statement — cannot be generated safely at test time, because a generator that changes meaning produces a failure you will spend an afternoon on. Write them once, have a human confirm each one means the same thing, and commit them as data.

// tests/data/refund_paraphrases.json
{
  "late_delivery": {
    "label": "approve",
    "variants": [
      "My order was supposed to arrive on the 3rd and it still hasn't come.",
      "It's past the delivery date and the parcel never showed up.",
      "Order NL-004417 is late. Delivery was promised for the 3rd; nothing arrived.",
      "I haven't received my package and the promised date has passed."
    ]
  }
}

Committing them as data rather than as code means the set is reviewable by somebody who is not a programmer, which is the person most likely to spot that variant three quietly added an order id the others do not have. That is exactly the kind of drift that produces a false failure.

Separating paraphrase variance from sampling variance

This is the step that is almost always missing. If you run each paraphrase once and two of them disagree, you do not know whether the rewording caused it or the sampler did. The test has no way to attribute its own failure, so it will be blamed on the prompt and somebody will spend a day rewriting a prompt that was fine.

The control is to measure the disagreement rate under a fixed input first. Run the original prompt k times, count how often the extracted label differs, and that is your floor. If the original disagrees with itself two times in ten, a paraphrase set that disagrees two times in ten has demonstrated nothing.

Do not expect that floor to be zero even at temperature zero. Greedy decoding picks the highest-scoring token, but the scores themselves are computed in floating point on batched hardware, and a different batch composition changes the order of the reductions that produce them. Two near-tied tokens can swap. This is a property of how inference is served, not a bug you can configure away, and it is why consistency tests of every kind need a measured baseline rather than an assumed one.

The baseline also tells you how many variants you need. If the fixed input agrees with itself unanimously over ten runs, four variants are enough to make a disagreement meaningful. If it disagrees with itself once in ten, four variants will produce a spurious failure roughly a third of the time you run the suite, and no amount of prompt work will fix that — the test is measuring the sampler. Either drive the floor down first or accept a majority threshold rather than unanimity, and write down which you chose and why next to the assertion.

The assertion

Two workable forms. The strict one asserts every variant matches a committed expected label, which is right when you are confident in the label. The relative one takes the majority label across the variant set and asserts agreement with it, which is right when you care about stability rather than correctness and do not want to maintain labels.

import json, pathlib, collections, pytest

CASES = json.loads(pathlib.Path("tests/data/refund_paraphrases.json").read_text())

@pytest.mark.parametrize("case_id", list(CASES))
def test_paraphrases_agree(case_id):
    variants = CASES[case_id]["variants"]
    labels = [classify(v)["decision"] for v in variants]
    counts = collections.Counter(labels)
    majority, n_majority = counts.most_common(1)[0]

    deviants = [
        (v, l) for v, l in zip(variants, labels) if l != majority
    ]
    assert not deviants, (
        f"{case_id}: {n_majority}/{len(labels)} agreed on {majority!r}; "
        + "; ".join(f"{l!r} <- {v!r}" for v, l in deviants)
    )

The failure message is doing real work there. “3/4 agreed on approve; ‘review’ came from the variant that mentions the order id” is a bug report. “assert False” is not. Print the deviant variants and their labels, never a count alone.

Running it

1. Pick five to ten cases with unambiguous answers and four to six variants each. That is 20–60 calls per run; decide now whether that runs on every pull request or nightly.
2. Measure the fixed-input baseline once, with a throwaway script that calls the original prompt k times and prints the label distribution. Record the number in a comment next to the test. If it is not close to unanimous, fix that before writing the paraphrase assertion at all.
3. Set temperature to zero and pin the model version. Both reduce the floor you just measured, and neither eliminates it.
4. Add the mechanical variants as a separate property test over generated whitespace and ordering changes, so a failure there is immediately distinguishable from a failure on a hand-written paraphrase. They are different bugs.
5. When a variant deviates, read it before touching the prompt. Most first failures are a variant that does not mean what the others mean.