Skip to content
Multigrid

Synthetic data & dataset curation

How to generate, filter, deduplicate and document a dataset — and what the published research actually says about training on data a model produced.

Every dataset is curated. The only question is whether anybody decided how. A corpus assembled by scraping whatever was reachable, labelled by whoever was available, and shipped without a duplicate check is a set of decisions too — they were just made by accident, and they show up later as a benchmark score that will not reproduce or a model with a verbal tic nobody can explain.

These pages are about making those decisions on purpose. Half of them are about generation: how to get examples out of a model, how to tell the good ones from the filler, and what happens when generated data feeds back into training. The other half are about the far less glamorous work that decides whether a dataset is worth anything — deduplication, decontamination, label-noise detection, provenance and documentation. The generation half gets the attention. The curation half is where the quality comes from.

Synthetic Data: What It's Good For and What It Isn't

The three unrelated things the term covers, and every use case sorted by how closely the generated distribution has to match the real one.

5 min read

Generating Training Data With an LLM: The Pipeline

A five-stage generation pipeline with a quality gate at every stage, the published recipes each stage comes from, and the arithmetic for what a run costs per surviving example.

7 min read

Model Collapse: What the Research Actually Showed

The Nature result, the follow-ups that reversed its conclusion, and the follow-ups that reversed it back — with the experimental regime each one holds under.

6 min read

Diversity in Synthetic Data: The Mode Collapse Problem

Four diversity metrics you can compute on your own corpus in an afternoon, what each one catches that the others miss, and why conditioning beats temperature.

6 min read

Synthetic Data for Testing and QA

Building realistic fixtures with seeded generators, referential integrity and the awkward cases a production dump would have given you for free.

5 min read

Privacy-Safe Synthetic Data From Real Records

What the published attack literature found about re-identification from synthetic records, why outliers are the failure case, and what differential privacy does and does not buy.

6 min read

Building an Eval Set From Scratch

How to assemble fifty examples that answer a question about your own system, and the binomial arithmetic that says exactly what fifty examples can and cannot resolve.

5 min read

Adversarial Examples for Robustness Testing

Generating the inputs that break your feature: invariance and directional tests, the perturbation catalogue, and the published methodologies each one comes from.

6 min read

Data Augmentation for Text: What the Studies Found

Paraphrase, back-translation, token noise and LLM rewriting — the published results for each, and the data regime where the gains were concentrated.

6 min read

Curating a High-Quality Instruction Dataset

The published quality-over-quantity results — LIMA, AlpaGasus, the textbook-quality work — what each actually compared, and a filter stack you can run over an existing set.

5 min read

Deduplicating and Decontaminating a Dataset

Exact and near-duplicate detection with MinHash and suffix arrays, n-gram decontamination against your eval sets, and what the deduplication studies reported.

6 min read

Human Preference Data: Collecting It Cheaply

Why pairwise comparison beats rating scales, the interface decisions that decide your agreement rate, and how to compute inter-annotator agreement on your own data.

5 min read

Active Learning: Labelling Only What Matters

Uncertainty sampling and its alternatives, the break-even arithmetic against random sampling, and the published warning about datasets acquired this way.

6 min read

Weak Supervision and Programmatic Labelling

Writing labelling functions instead of labels, how a label model reconciles them without ground truth, and where the approach stops working.

6 min read

Dataset Documentation: Datasheets That Get Used

The datasheet framework, the handful of fields that actually get read six months later, and a template short enough that people fill it in.

5 min read

Licensing and Provenance of Training Data

Tracking where every row came from and under what terms, why aggregator licence tags are unreliable, and a provenance record you can attach at ingest.

5 min read

Detecting Label Noise in an Existing Dataset

Confident learning, cross-validated out-of-sample probabilities, and the published audit that found label errors in the test sets of ten standard benchmarks.

5 min read

Balancing a Dataset Without Throwing Data Away

Why resampling is usually the wrong tool, what it does to calibrated probabilities, and the decision rule for choosing between weighting, thresholding and collecting more.

5 min read

Domain-Specific Corpora: Building One for Your Industry

Where domain text actually comes from, the quality-classifier approach to filtering it, and the licensing questions that decide whether you can use any of it.

6 min read

When Synthetic Data Fails: Five Failure Modes and Their Signatures

The five ways a generated dataset goes wrong, the published evidence for each, and the specific check that catches it before training rather than after.

6 min read

Other topics

Synthetic data & dataset curation · Multigrid