Synthetic Data for Testing and QA

Almost every team starts with a sanitised copy of production, and almost every team eventually stops. The interesting question is what has to be true of the replacement before anyone will actually use it.

Why the production dump has to go

The habit is understandable. A production copy is realistic by construction, it has the awkward rows in it, and it costs one command. The problems accumulate rather than arriving at once:

• Sanitising is not anonymising. Replacing names and emails leaves the structure — the join graph, the timestamps, the amounts, the rare combinations — and structure re-identifies. Redaction reduces risk and does not eliminate it, and the residual is hard to argue about with a regulator.
• It ages. The dump is from March. The schema moved in May. Now every developer has a local fix-up script and no two are the same.
• It cannot express a case that has not happened yet. You cannot write a test for the refund-after-partial-shipment path until a customer has done it.
• It is enormous. Nobody runs the suite against forty gigabytes on a laptop, so people test against a subset that was chosen by LIMIT 1000 and contains none of the interesting rows.
• It puts real customer data on developer machines. Which is a sentence that ends most discussions once someone writes it down in a data-protection assessment.

What a fixture generator must reproduce

A generator that does not satisfy these will be abandoned within a quarter, because a test that fails intermittently is worse than no test. The list is short and none of it is optional; a generator missing any one row here fails in a way that gets blamed on the test suite rather than on the fixtures, which is why it is worth reading as a specification rather than as advice.

A seeded generator with referential integrity

The shape matters more than the library. Everything derives from one seed, parents are generated before children, and the id lists are carried forward rather than reconstructed.

import random
from datetime import datetime, timedelta
from faker import Faker

def build_dataset(seed: int = 20260803, n_customers: int = 500):
    """One seed drives everything. Same seed in, same bytes out."""
    rng   = random.Random(seed)
    fake  = Faker()
    Faker.seed(seed)                      # faker has its own RNG — seed it too
    epoch = datetime(2026, 1, 1)

    customers, orders, refunds = [], [], []

    for cid in range(1, n_customers + 1):
        signup = epoch + timedelta(days=rng.randint(0, 500))
        customers.append(dict(
            id=cid, name=fake.name(), email=fake.email(),
            country=rng.choices(["NL","DE","US","JP"], weights=[5,3,3,1])[0],
            signed_up_at=signup,
        ))

        # Skewed cardinality: most customers order little, a few order a lot.
        n_orders = rng.choices([0, 1, 2, 5, 40], weights=[40, 30, 20, 9, 1])[0]
        for _ in range(n_orders):
            placed = signup + timedelta(days=rng.randint(0, 400))   # after signup
            oid = len(orders) + 1
            orders.append(dict(
                id=oid, customer_id=cid, placed_at=placed,
                amount_cents=rng.choice([99, 1250, 4999, 129900]),
                status=rng.choices(["paid","shipped","cancelled"],
                                   weights=[5, 4, 1])[0],
            ))
            if rng.random() < 0.05:                      # refunds are rare
                refunds.append(dict(
                    order_id=oid,                        # FK always resolves
                    refunded_at=placed + timedelta(days=rng.randint(1, 60)),
                    amount_cents=orders[-1]["amount_cents"],
                ))

    return dict(customers=customers, orders=orders, refunds=refunds)

Two details that are easy to get wrong. Seed every random source, not just the one you remembered — a fake-data library keeps its own generator, and an unseeded one makes the whole dataset non-reproducible. And derive child timestamps from the parent’s rather than from the epoch, which is what keeps “refunded after ordered” true without a validation pass.

The catalogue of nasty rows

This is the part that repays being explicit, because it converts “the dump happened to contain a customer with an apostrophe in their name” into a reviewed list that grows every time production teaches you something. Append to it after every incident.

• Text that breaks assumptions. Apostrophes and quotes, right-to-left scripts, four-byte emoji, combining characters, zero-width joiners, a name that is one character long and one that is 400, leading and trailing whitespace, a string that looks like a number, a string that looks like JSON.
• Numbers at the boundary. Zero, negative, the largest value the column can hold, an amount with more decimal places than the currency has, a currency that does not have two decimal places at all.
• Time. The 29th of February, a timestamp inside the daylight-saving gap, a record created and updated in the same millisecond, a future date, a date before the company existed, two events in different time zones that must be ordered.
• Cardinality extremes. The customer with zero orders, the customer with ten thousand, the order with one line item and the one with three hundred. These find the pagination bugs and the queries that were only ever run against small rows.
• Legitimately incomplete rows. Optional fields that are genuinely absent, a soft-deleted parent with live children, an account mid-migration. Real data is full of states your schema permits and your code does not expect.

Where property-based testing takes over

A fixture generator produces one dataset per seed. A property-based testing library — Hypothesis in Python, fast-check in TypeScript, QuickCheck’s descendants elsewhere — produces a family of datasets and searches it for a counterexample, then shrinks that counterexample to the smallest failing case. That last step is what makes it a different tool rather than a fancier one: it hands you a minimal reproduction instead of a 200-row dataset that fails somewhere.

The division of labour that works: use the seeded generator for fixtures a human reads and a demo shows, and property-based generation for invariants — the balance never goes negative, a refund never exceeds its order, serialising and parsing round-trips. Both are synthetic data. Only one of them is trying to look real.

There is a third role worth separating out, because it is the one that quietly justifies the whole investment: the generator becomes the fastest way to construct a state that is hard to reach through the product. Reproducing a bug that only appears for an account with a lapsed subscription, two currencies and a partially refunded order takes twenty minutes of clicking and one line of generator configuration. Once a team notices that, the generator stops being a test-data chore and becomes a debugging tool, and it starts getting maintained for the same reason any tool people rely on does.