Diversity in Synthetic Data: The Mode Collapse Problem

Long before anything resembling model collapse, a generated dataset fails in a duller and far more common way: it is repetitive. The good news is that repetition is one of the few dataset properties that is genuinely cheap to measure.

What the failure looks like

Ask a model for a thousand customer-support scenarios and you get a thousand documents. Read forty of them and you find perhaps six underlying situations, three recurring first names, one opening clause that appears in a third of the corpus, and a length distribution so tight it looks like a specification was enforced. Nothing is wrong with any individual row. The corpus is still nearly worthless, because a training set of a thousand rows expressing six ideas is six ideas with expensive noise attached.

This is the same phenomenon that mode collapse named in the GAN literature — a generator concentrating its mass on a few modes of the target distribution — and it arrives here without any adversarial training involved, simply because sampling a language model returns its high-probability regions far more often than its tails. It is also the first symptom of the tail loss that the collapse literature describes, which makes it worth monitoring for two reasons rather than one.

Four metrics, four blind spots

None of these is sufficient alone, and the reason to run all four is that each is blind to a failure the next one catches.

It helps to separate two things the word diversity is doing here. Lexical diversity is about how the corpus is worded: whether the same phrases recur, whether every response opens the same way, whether one construction dominates. Semantic diversity is about how many distinct situations the corpus actually contains. The two come apart constantly, and always in the same direction — a generator asked for variety will produce lexical variety readily and semantic variety only when the conditioning forces it. A corpus can therefore score respectably on the first two metrics below and still be six ideas wearing a thousand costumes, which is why the third one is on the list at all.

Computing all four

import random
from collections import Counter
import numpy as np
from sklearn.cluster import KMeans

def distinct_n(texts, n=3):
    """Unique n-grams / total n-grams, corpus-wide. Range (0, 1]."""
    total, uniq = 0, set()
    for t in texts:
        toks = t.split()
        grams = list(zip(*[toks[i:] for i in range(n)]))
        total += len(grams)
        uniq.update(grams)
    return len(uniq) / max(total, 1)

def self_bleu(texts, sample=300, seed=0):
    """Average BLEU of each doc against a random other doc. Lower = diverse.
    Sampled pairs, because the full computation is quadratic."""
    from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction
    rng = random.Random(seed)
    sm = SmoothingFunction().method1
    scores = []
    for _ in range(sample):
        a, b = rng.sample(texts, 2)
        scores.append(sentence_bleu([b.split()], a.split(), smoothing_function=sm))
    return sum(scores) / len(scores)

def cluster_coverage(embeddings, k=50, mass=0.90, seed=0):
    """How many clusters hold 'mass' of the corpus? Higher = more spread out."""
    labels = KMeans(n_clusters=k, random_state=seed, n_init=10).fit_predict(embeddings)
    sizes = sorted(Counter(labels).values(), reverse=True)
    running, needed = 0, mass * len(labels)
    for i, s in enumerate(sizes, start=1):
        running += s
        if running >= needed:
            return i          # e.g. 6 of 50 clusters hold 90% => concentrated
    return k

def openings(texts, n=5, top=10):
    """The verbal tic detector. Prints the most common first-n-token prefixes."""
    c = Counter(" ".join(t.split()[:n]) for t in texts)
    return c.most_common(top)

def length_stats(texts):
    lens = np.array([len(t.split()) for t in texts])
    return dict(mean=lens.mean(), sd=lens.std(),
                p10=np.percentile(lens, 10), p90=np.percentile(lens, 90))

Run openings() first. It takes two seconds, needs no dependencies, and in a bad corpus it will show you that a quarter of the rows begin with the same five words — which is the finding that actually changes what you do next, and it arrives before you have embedded anything.

The number is meaningless without a baseline

A distinct-3 of 0.42 is neither good nor bad. Every one of these metrics is sensitive to corpus size, to document length, to tokenisation and to domain, so an absolute threshold is not transferable between two datasets and certainly not between two papers. There are exactly two comparisons worth making:

• Against real data of the same size and domain. Take an equally sized sample of human-written text from the same task and compute the identical metric. The gap is the finding. If your generated corpus has half the distinct-3 of the real sample, you know something specific.
• Against yesterday’s run. Once the metric is in the pipeline, it becomes a regression test: a prompt change that halves cluster coverage is caught the same day rather than after training. This is the higher-value use of the two, because it costs nothing per run.

There is a third comparison people reach for and should not: the numbers in a paper. Published diversity figures were computed on a different corpus, at a different size, with a different tokenisation and often a different definition of the same metric — self-BLEU in particular varies with how many reference documents each comparison uses. Treating a number from a benchmark as a target produces a pipeline optimised for a quantity that does not describe your data.

Subsample carefully when comparing. Distinct-n falls as a corpus grows, simply because the space of n-grams is finite, so comparing a 10,000-row generated set against a 500-row human one measures the size difference and nothing else.

Conditioning beats temperature

The instinctive fix is to raise the sampling temperature. It is the weakest lever available, and it degrades quality on the way: high temperature buys you lexical variation around the same content, which moves distinct-n and leaves cluster coverage exactly where it was. That is the honest test of whether a diversity intervention did anything — if the semantic metric did not move, you have relabelled the problem.

What moves the semantic metric is entropy in the conditioning:

• Enumerate the axes. Domain, persona, difficulty, document type, locale, failure mode. Walk the cross-product rather than sampling it. Five axes of four values each is 1,024 distinct contexts, and the generator cannot collapse across contexts it never sees together.
• Seed from real artefacts. A generation conditioned on a real document, a real ticket or a real row inherits that artefact’s variety for free. This is the highest-yield intervention available and it requires no prompt engineering at all.
• Apply explicit difficulty operators. The Evol-Instruct approach — rewriting an instruction to add constraints, deepen it or increase its reasoning steps — produces a spread along an axis that sampling does not reach.
• Vary the generator. Two models produce two priors. Splitting a run across model families is a diversity intervention that costs nothing but a config change, and it also breaks the single-generator failure mode where every systematic error in the corpus has one source.
• Filter for novelty, not just quality. A near-duplicate check inside the generation loop — reject a candidate whose similarity to anything already accepted exceeds a threshold — is the Self-Instruct move, and it enforces diversity structurally rather than hoping for it.