Data Augmentation for Text: What the Studies Found

Text augmentation has a genuine research record behind it, and the most useful thing in that record is not which technique wins. It is the repeated finding about when any of them helps at all.

The finding that decides whether to bother

Wei and Zou introduced EDA — Easy Data Augmentation — in 2019: four trivially simple operations (synonym replacement, random insertion, random swap, random deletion) applied to training sentences. Across five text-classification benchmarks they reported average gains that were small on full training sets and substantially larger when only a few hundred labelled examples were available. The headline they drew from it is the one to remember: augmentation is a small-data technique.

That pattern recurs. Xie and colleagues’ Unsupervised Data Augmentation for Consistency Training (2019/2020) reported its most striking results in extreme low-label settings — a sentiment classifier trained with a couple of dozen labelled examples plus large unlabelled data augmented by back-translation, competitive with models trained on the full labelled set. The mechanism there is consistency training rather than plain augmentation, but the regime is the same: augmentation buys the most where labels are scarcest.

The practical rule that falls out: if you already have tens of thousands of labelled examples covering your task, augmentation is unlikely to be the intervention that moves your numbers, and the same effort spent on finding the mislabelled rows or on deduplication usually pays better. If you have three hundred, augmentation is one of the cheapest things you can try.

The four techniques and their evidence

Token-level noise

Synonym replacement, random insertion, random swap, random deletion — EDA’s four. Cost: nothing, no model call, milliseconds per example. Risk: synonym replacement from a static thesaurus changes meaning more often than people expect, and it is exactly the words that carry the label that a thesaurus handles worst. Best used with a low replacement rate and never on the span that determines the label.

Back-translation

Translate to another language and back. The technique comes from machine translation, where Sennrich and colleagues (2016) used back-translated monolingual data to improve neural MT, and it transferred to augmentation because the round trip produces a genuine paraphrase rather than a word-swap. It preserves meaning far better than token noise and produces more varied syntax. Cost is two translation calls per example, and the pivot language matters: a distant language gives more variation and more drift.

Paraphrase by model

Ask a language model to rewrite the example. Highest quality, highest cost, and the one that most needs a diversity check — a model asked for five paraphrases will often produce five rewordings with an identical structure, which adds tokens and no information. Run the diversity metrics on the augmented set before training on it, comparing against the originals.

Structural and template augmentation

Change what surrounds the content rather than the content: reformat as a bullet list, change the greeting, convert a question to an imperative, wrap the same fields in a different JSON shape. For extraction and classification tasks this is often the highest-yield family, because it targets the exact invariance you want the model to have — and it overlaps directly with the invariance tests in robustness testing. Augmentation and robustness testing are the same perturbations pointed at training and at evaluation respectively.

The rule that keeps augmentation honest

One rule prevents most augmentation disasters: a perturbation that can change the label is not augmentation, it is mislabelling.

Random deletion applied to “the refund was not approved” can delete the negation and produce a training example whose label is now wrong. Synonym replacement on a medical dosage, an entity name or a legal term does the same thing more quietly. The damage is worse than adding no data at all, because you have added confidently mislabelled data that the training loop will fit.

Three defences, in order of strength:

• Protect the label-bearing span. If you know which substring determines the answer — the extracted value, the negation, the entity — exclude it from perturbation entirely. This is mechanical and it eliminates most of the risk.
• Verify after augmenting. Where a verifier exists, run it on the augmented example. For extraction: does the target value still appear verbatim in the perturbed document? If not, drop the row.
• Read a sample. Fifty augmented examples, by eye, before training on eighty thousand. This finds the systematic problems in ten minutes, and systematic is what augmentation errors always are.

An augmentation pass with a label guard

import random

PROTECTED = ("not", "no", "never", "except", "unless", "without")

def synonym_swap(text, rng, thesaurus, rate=0.1):
    """Never touch a negation, and never touch the protected span."""
    words = text.split()
    n = max(1, int(len(words) * rate))
    idx = [i for i, w in enumerate(words)
           if w.lower() not in PROTECTED and w.lower() in thesaurus]
    for i in rng.sample(idx, min(n, len(idx))):
        words[i] = rng.choice(thesaurus[words[i].lower()])
    return " ".join(words)

def augment(example, rng, thesaurus, k=2):
    """Augment ONLY the training split, and keep provenance on every row."""
    out = []
    for _ in range(k):
        text = synonym_swap(example["text"], rng, thesaurus)
        if example["label_span"] and example["label_span"] not in text:
            continue                      # guard: the label evidence survived?
        out.append(dict(text=text, label=example["label"],
                        origin=example["id"], synthetic=True))
    return out

def build(train, dev, test, seed=0):
    rng = random.Random(seed)
    thesaurus = load_thesaurus()
    augmented = [r for ex in train for r in augment(ex, rng, thesaurus)]
    # dev and test are NEVER augmented, and no augmented row may share an
    # origin with anything in dev or test.
    origins = {r["origin"] for r in augmented}
    assert not (origins & {e["id"] for e in dev + test})
    return train + augmented, dev, test

The assertion at the end is the one people skip and regret. Augmenting before splitting puts paraphrases of the same example on both sides of the split, and the evaluation score that results is measuring memorisation. Split first, augment the training half only, and check that no augmented row traces back to an example in dev or test.

What changed when the generators got good

Most of the augmentation literature predates instruction-following models being cheap, and two things genuinely shifted.

First, the boundary between augmentation and generation dissolved. Asking a model to paraphrase an example and asking it to write a new example are the same API call with a different prompt, so the question is no longer “which augmentation operator” but how the generation pipeline is gated. Augmentation is now a special case of generation in which the seed is a labelled example — which is a good special case to be in, because the label comes along for free.

Second, the reason to augment moved. In the small-data classifier era, augmentation was about regularisation: more variety, less overfitting. When the downstream step is fine-tuning a large pretrained model, the model has already seen a great deal of language and does not need to be taught that word order varies. What it needs is coverage of situations it has not seen in your task. That makes structural and scenario augmentation — new conditions, new document layouts, new edge cases — worth far more than lexical noise, and it is why the token-level operators feel much less useful now than the papers make them sound.