Adversarial Examples for Robustness Testing

Robustness testing asks a narrower question than red teaming: not “can this be made to misbehave” but “does the answer change when it should not”. That question has a well-developed methodology behind it, and it generates far more actionable failures.

This is not the security page

Two different activities share the word adversarial. Security testing is about an attacker with a goal — extracting a system prompt, causing a tool call, exfiltrating data — and it belongs with prompt injection and red teaming. Robustness testing is about a normal user with an unusual input: a typo, a dialect, a longer document, a name the tokeniser splits oddly, a question phrased in the passive voice.

Robustness failures are more common, they hit real users rather than hypothetical attackers, and — the part that makes them worth generating in bulk — they come in families. One perturbation applied across a hundred existing eval examples produces a hundred new test cases whose expected outputs you already know, which is the trick that makes this cheap.

Three test types that cover most of it

Ribeiro and colleagues set out this taxonomy in Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (ACL 2020), and it transfers to language-model features essentially unchanged. The value is that each type tells you what the expected output is without anyone labelling anything.

Invariance tests are the highest-yield of the three by a wide margin, for one structural reason: they need no ground truth. Any existing input becomes a test case the moment you perturb it, so an eval set of fifty examples and a catalogue of ten perturbations is five hundred assertions that cost nothing to label.

Two related methodologies are worth knowing. Contrast sets (Gardner et al., 2020) ask annotators to make minimal edits to existing examples that flip the correct label, which probes the local decision boundary rather than the average case. And dynamic adversarial collection — the approach behind Adversarial NLI (Nie et al., 2020) and the Dynabench platform (Kiela et al., 2021) — has humans write examples against a model in the loop, keeping the ones that fool it, then retraining and repeating.

The perturbation catalogue

Grouped by what they are testing for. Start with the first group; it is where the failures are.

• Surface noise (invariance expected). Character transposition and doubling, missing accents, ALL CAPS, no capitalisation at all, missing punctuation, a trailing newline, double spaces, smart quotes instead of straight ones, a zero-width character somewhere in the middle.
• Lexical substitution (invariance expected). Swap entity names, swap the locale, swap a product name for another of the same kind, replace a formal register with a casual one, translate to a second language and back.
• Structural (invariance expected). Reorder independent bullet points, move the question from the start of the prompt to the end, wrap the same content in Markdown instead of plain text, add an irrelevant preamble, add an irrelevant trailing sentence.
• Semantic (direction expected). Insert a negation, change a quantity by an order of magnitude, add a condition that should trigger a refusal, change a date so a deadline has passed, introduce a contradiction between two parts of the input.
• Distribution shift (behaviour change expected, degradation not). A document three times longer than any in your eval set, a language your traffic contains but your tests do not, a scanned page instead of a clean one, an input that is entirely whitespace.
• Position and length. Move the critical fact from the top of a long context to the middle. That is the specific failure long-context degradation describes, and it is trivially generated by padding.

Generating the perturbed set

Deterministic perturbations first — they are seedable, free and reproducible. Reach for a model only for the ones that require meaning.

import random, re

def typo(text, rng):
    """Swap two adjacent characters in a random word of 4+ letters."""
    words = text.split()
    idx = [i for i, w in enumerate(words) if len(w) >= 4]
    if not idx: return text
    i = rng.choice(idx)
    w = list(words[i]); j = rng.randrange(len(w) - 1)
    w[j], w[j+1] = w[j+1], w[j]
    words[i] = "".join(w)
    return " ".join(words)

def strip_punctuation(text, rng):  return re.sub(r"[.,;:!?]", "", text)
def shout(text, rng):              return text.upper()
def add_preamble(text, rng):       return "Hi there, quick one — " + text
def add_trailer(text, rng):        return text + "\n\nThanks in advance!"
def zero_width(text, rng):
    i = rng.randrange(len(text));  return text[:i] + "\u200b" + text[i:]

INVARIANT = [typo, strip_punctuation, shout, add_preamble, add_trailer, zero_width]

def invariance_suite(examples, seed=0, per_example=3):
    """Each perturbed input must produce the SAME output as its original.
    No labelling required: the original output is the expected value."""
    rng = random.Random(seed)
    cases = []
    for ex in examples:
        baseline = run_feature(ex["input"])
        for fn in rng.sample(INVARIANT, per_example):
            cases.append(dict(
                original=ex["input"],
                perturbed=fn(ex["input"], rng),
                perturbation=fn.__name__,
                expected=baseline,
            ))
    return cases

def report(cases):
    """Group failures BY PERTURBATION. One broken perturbation across many
    examples is one bug; the same count spread evenly is a different bug."""
    from collections import Counter
    fails = Counter(c["perturbation"] for c in cases
                    if run_feature(c["perturbed"]) != c["expected"])
    return fails.most_common()

The reporting function is the part that is easy to omit and repays keeping. A robustness run produces hundreds of failures and they are not independent: if zero_width accounts for ninety of them, you have one input-sanitisation bug, not ninety model problems. Grouping by perturbation turns a wall of red into a short list.

Where humans still beat generators

A generated adversarial set is bounded by the imagination encoded in the catalogue. Three categories that reliably require a person:

• Domain-specific traps. The clause that means something different in insurance than in ordinary English; the product configuration that is technically orderable and never shipped. Only somebody who does the job knows these exist.
• Genuinely ambiguous inputs. Cases where two competent people disagree about the correct answer. These are the most valuable test cases you can own, and no generator produces them because a generator has no idea it is being ambiguous.
• Real user weirdness. Nothing invented matches what actual people send. Sampling the strangest 1% of production inputs costs one query and outperforms an afternoon of invention.

What to do with a failure

Not every robustness failure is a bug worth fixing, and treating them as equivalent is how a robustness suite gets switched off. Sort them:

• Fix in code, not in the prompt. Whitespace, control characters, encoding, casing and length limits belong in input normalisation. A model should never be the thing defending you against a zero-width space.
• Fix in the prompt. Failures on negation, on conditions and on ordering usually mean the instruction is underspecified rather than the model is weak.
• Accept and record. Some perturbations genuinely should change the output, and your invariance assumption was wrong. Move the case to a directional test and note why.
• Promote to the eval set. Anything that broke and mattered joins the permanent eval set so it stays fixed. That promotion path is what makes robustness testing cumulative rather than an event.