Semantic Diff for Comparing Two Model Outputs in Tests

You have two generations of the same prompt — yesterday’s golden file and today’s output, or one model’s answer and another’s — and you need a number that says whether the difference matters. The instinct is to reach for a similarity score. The useful move is to decide, before you score anything, which layer of difference you are actually asking about.

Why a character diff is the wrong instrument

The standard library diff you already have is a character or line diff. Python’s difflib.SequenceMatcher implements a Ratcliff/Obershelp-style longest-matching-block ratio; Git’s diff is Myers on lines. Both answer “how much of this text is literally the same text”, and that question is neither sound nor complete for meaning.

It is not complete because two texts can mean the same thing and share almost no characters. “The refund will reach you in 3–5 working days” and “Refunds take three to five business days to arrive” score badly on any character measure and are the same answer. It is not sound because two texts can differ by three characters and mean opposite things: approved versus not approved. A test that passes the second pair and fails the first is exactly backwards, and it will fail that way on the day somebody rewords a system prompt.

This page is about scoring meaning-level difference. It is a different job from the cost-oriented prompt diff that shows you which tokens changed between two prompt versions and what that did to your bill; that tool answers “what did I change”, and this one answers “did the answer change”.

Four layers, cheapest first

Work down this list and stop at the first layer that can answer your question. Most teams jump to layer three and never notice that layer one would have done it exactly, for free, and deterministically.

• Structural. Parse both outputs into fields and diff the fields. If the model returns JSON, or you can extract a label, an amount, a tool name and a set of cited ids, then the comparison is equality on those values and there is no similarity score anywhere. This is the layer to fight for: it is exact, it is stable, and its failures name the field that broke. See snapshotting the parsed object rather than the string.
• Normalised lexical. Lowercase, collapse whitespace, strip trailing punctuation, optionally sort tokens, then compare. This catches formatting churn and nothing else. It is worth one line of code as a pre-filter: if the normalised forms are identical, no further layer needs to run.
• Embedding similarity. Embed both texts and take the cosine. This is what most people mean by “semantic diff”, and it is the layer with the worst failure mode: sentence embeddings place a claim and its negation very close together, because they share almost all of their content words. If negation is a possible regression in your output — and for anything that makes a decision it is — cosine alone will not catch it.
• Entailment. Ask a natural-language-inference model in both directions. If A entails B and B entails A, they are paraphrases; if either direction gives contradiction, you have a real regression. This is the only layer that gets negation right, and it is also the slowest and the one that introduces a second model into your test.

Between layers three and four sits BERTScore, published by Zhang, Kishore, Wu, Weinberger and Artzi at ICLR 2020 (BERTScore: Evaluating Text Generation with BERT). It greedily matches tokens between the two texts in contextual embedding space and reports precision, recall and F1 rather than one blended number, which is more informative than a sentence cosine because you can see whether the new output dropped content or added it. Its raw scores compress into a narrow high band, which is why the reference implementation offers baseline rescaling; if you skip that, every pair will look similar and your threshold will be meaningless.

Picking a threshold you can defend

The number in assert score > 0.85 is the whole test, and 0.85 is almost always chosen because it looks like a lot. Derive it instead. Take twenty pairs you have already labelled “same answer” and twenty labelled “different answer”, score all forty, and pick the value that separates them. Commit the labelled pairs next to the test. Now the threshold is a derived constant with a re-runnable derivation, and when somebody swaps the embedding model the fix is to re-derive rather than to argue.

Two things follow from that. The first is that the threshold belongs to the embedding model, so the model identifier has to be pinned and recorded in the test output; an unpinned hosted embedding endpoint silently moves every score in your suite at once. The second is that a well-separated corpus is itself the finding: if your “same” and “different” pairs overlap in score, no threshold exists, and the honest conclusion is that this layer cannot answer your question and you need the layer below it.

Your question is directional; cosine is not

A regression test rarely asks “are these the same”. It asks “is the new one still acceptable”, and that is a directed question. Similarity metrics are symmetric, so they cannot express it. Splitting the output into three classes of field usually can:

• Must be identical. The decision, the order id, the amount, the tool name, the set of cited document ids. Assert equality. No score.
• Must be equivalent. The explanation attached to the decision. Assert entailment in both directions, or a derived similarity threshold if entailment is too expensive to run per test.
• Free to vary. Greeting, ordering of a bulleted rationale, the exact wording of a sign-off. Assert nothing, and delete these from the golden file so they cannot cause a diff at all.

That third bullet does more work than any scoring choice. Most “semantic diff” problems are really the problem that the artefact under comparison contains fields nobody ever intended to assert on.

What the failure message has to contain

A diff test fails in CI at 02:00 and somebody has to decide in ninety seconds whether to approve the change or revert it. The assertion message is the entire interface to that decision, so it must carry: the layer that failed, both texts in full, the score and the threshold, the identifier of whatever model produced the score, and the field name if the failure was structural.

def assert_semantically_equal(expected: str, actual: str, *, threshold: float) -> None:
    if normalise(expected) == normalise(actual):
        return
    score = cosine(embed(expected), embed(actual))
    if score < threshold:
        raise AssertionError(
            f"semantic diff below threshold\n"
            f"  scorer:    {EMBEDDING_MODEL_ID}\n"
            f"  score:     {score:.3f} (threshold {threshold:.3f})\n"
            f"  expected:  {expected!r}\n"
            f"  actual:    {actual!r}"
        )

The reviewer’s next step after reading that message is a decision, not an investigation, and the workflow around approving or rejecting the diff is where the actual cost of this kind of test lives. Scoring is the easy half.