Synthetic Tabular Data Generation for Testing a Pipeline

Testing a feature pipeline needs a table that looks like production: the same columns, the same dtypes, plausible marginals, and correlations that do not collapse the moment a downstream step computes a ratio. Random noise fails that; a copula sample passes it, and takes about forty lines.

What synthetic data is actually for here

Distinguish two goals that get conflated, because the bar is completely different. Privacy-preserving release means the synthetic table can be published in place of the real one, which requires a formal privacy argument, an attack model, and usually differential privacy with an accounted epsilon. This page is not about that, and a copula fitted to a small table can memorise individual rows.

Pipeline testing means the synthetic table exercises your code the way the real one does: joins produce the right cardinalities, aggregations produce non-degenerate values, a correlation-based redundancy check finds the pairs it is supposed to find, and an imputation step meets nulls in the columns that actually have them. That is a much lower bar, and a Gaussian copula clears it.

The distinction matters for a practical reason: if you are testing a pipeline, the synthetic data does not need to be good, it needs to be structurally faithful. A column whose mean is 12% off will not fail a test. A column whose correlation with another is 0 when it should be 0.8 will hide a bug in every feature that divides one by the other.

The Gaussian copula, in three steps

A copula separates the question of what each column’s distribution looks like from the question of how the columns move together. That separation is exactly what you want, because the marginals of a real table are usually ugly — skewed, bounded, spiky at zero — while the dependency structure is often adequately captured by a correlation matrix.

1. Transform each column to a uniform by its empirical rank: value → rank / (n + 1). This is the probability integral transform, done non-parametrically so it works on any shape.
2. Transform each uniform to a standard normal with the inverse normal CDF, then measure the correlation matrix of the resulting normals. This is the copula’s parameter.
3. Sample new rows from a multivariate normal with that correlation matrix, push them back through the normal CDF to uniforms, and map each uniform back to a value by taking the corresponding quantile of the original column.

The last step is what preserves the marginals exactly: every synthetic value is an interpolation between two real values from that column, so bounds, skew and spikes come along for free. And because ranks are used throughout, the correlation that is preserved is the rank correlation — Spearman, not Pearson — which is the right thing for skewed columns. This is the same construction that underlies the Gaussian copula synthesiser in the Synthetic Data Vault; Patki, Wedge and Veeramachaneni describe the approach in “The Synthetic Data Vault”, presented at IEEE DSAA in 2016.

The generator

numpy, pandas and scipy only. It handles numeric columns directly and categoricals by rank-coding them, which is discussed in the next section.

import numpy as np, pandas as pd
from scipy.stats import norm

def fit_copula(df: pd.DataFrame):
    """Returns everything needed to sample: the normal-space correlation
    matrix and the sorted values of each column."""
    n = len(df)
    z = pd.DataFrame(index=df.index)
    quantiles = {}
    for col in df.columns:
        s = df[col]
        # rank -> uniform in (0, 1), average ranks so ties do not stack at one point
        u = s.rank(method="average", na_option="keep") / (n + 1)
        z[col] = norm.ppf(u)
        quantiles[col] = np.sort(s.dropna().to_numpy())
    corr = z.corr(method="pearson").to_numpy()       # correlation of the normals
    corr = nearest_psd(corr)
    null_rate = df.isna().mean().to_dict()
    return corr, quantiles, null_rate, list(df.columns)

def nearest_psd(a, eps=1e-8):
    """Rank deficiency and NaN handling can push the matrix just off PSD."""
    vals, vecs = np.linalg.eigh((a + a.T) / 2)
    vals = np.clip(vals, eps, None)
    out = vecs @ np.diag(vals) @ vecs.T
    d = np.sqrt(np.diag(out))
    return out / np.outer(d, d)

def sample_copula(fitted, n_rows, seed=0):
    corr, quantiles, null_rate, cols = fitted
    rng = np.random.default_rng(seed)
    z = rng.multivariate_normal(np.zeros(len(cols)), corr, size=n_rows)
    u = norm.cdf(z)
    out = {}
    for i, col in enumerate(cols):
        q = quantiles[col]
        # map each uniform to the matching quantile of the real column
        idx = np.clip((u[:, i] * len(q)).astype(int), 0, len(q) - 1)
        vals = q[idx].astype(object)
        if null_rate[col] > 0:                        # restore the missingness rate
            vals[rng.random(n_rows) < null_rate[col]] = np.nan
        out[col] = vals
    return pd.DataFrame(out)

fitted = fit_copula(real_df.select_dtypes("number"))
fake = sample_copula(fitted, n_rows=50_000, seed=1)

print(real_df.corr(method="spearman").round(2))
print(fake.astype(float).corr(method="spearman").round(2))

Print the two Spearman matrices side by side and they should agree to within sampling error at 50,000 rows. That comparison is the acceptance test for the generator, and it is worth keeping as an assertion in the test suite so a future change to the fitting code cannot quietly produce independent columns.

Categoricals, nulls and constraints

For a categorical column, replace the rank transform with a mapping from category to its cumulative frequency interval, and the inverse with a lookup of which interval the uniform falls into. The order you assign the categories in decides which ones end up correlated with a high value of a partner column, so use the true order for an ordinal column and sort by target rate or by frequency for a nominal one. There is no correct answer for nominal, which is one of the honest limits of this method.

Missingness is restored above as an independent per-column rate, which is fine when values are missing at random and wrong when they are not. If nulls cluster — three columns from the same source system are null together — model the null pattern as its own categorical column and sample it alongside the values. Whether that matters depends on what the pipeline does downstream; an imputation step that treats each column independently will not notice, and one that uses a multivariate imputer will. See missing value imputation methods for which is which.

Column count deserves one note of its own. Fitting the copula requires a p×p correlation matrix and sampling requires a Cholesky-style factorisation of it, which is O(p³). At a few hundred columns that is instant; at several thousand it becomes the slow step, and the matrix is also more likely to be rank-deficient because you have fewer rows than the estimate needs. The nearest_psd repair above handles mild cases; for a genuinely wide table, fit the copula on a chosen subset of columns and generate the rest independently.

Hard constraints survive nothing. A copula does not know that end_date >= start_date, that a percentage lies in 0–100, or that three component columns sum to a total column. Enforce those after sampling with an explicit repair pass, and assert them in the test, because a pipeline that crashes on an impossible synthetic row teaches you nothing about production.

What it does not reproduce

• Non-monotone and conditional dependence. A Gaussian copula encodes pairwise, monotone association. A relationship that is U-shaped, or that only exists within one segment, is flattened to whatever pairwise rank correlation it implies.
• Tail dependence. The Gaussian copula is asymptotically independent in the tails: extreme values in two columns co-occur less often in the synthetic sample than in reality. If your pipeline is about extremes — fraud, outages, spikes — this is a serious limitation and a t-copula is the usual answer.
• Anything relational. One row per order with a plausible customer id, where the same customer must have consistent attributes across their rows, is a different problem. Generate the parent table first and sample children conditional on it.
• A defensible privacy guarantee. The stored quantilesare the real values, sorted. Do not ship the fitted object anywhere the raw data could not go.

Used inside the boundary those limits draw, it is a good tool: a fixed seed gives a reproducible fixture, the volume is free, and the structural properties your data quality checks assert on the real table hold on the synthetic one.