Synthetic Image Data for Training a Classifier

Synthetic images are not a cheaper substitute for real ones. They are good at a different thing — perfect labels and controllable rare cases — and treating them as a volume replacement produces a model that is excellent on renders and worse than the one you started with on photographs.

What synthetic data is actually good at

Three properties are genuinely unavailable from real data, and they are the honest reasons to render.

• The labels are exact and free. A renderer knows which pixels belong to which object, so segmentation masks, depth maps, surface normals and 6-DoF poses come out with no annotation cost and no annotator disagreement. For dense tasks this is the dominant argument: a pixel-accurate mask is expensive and slow to draw by hand and slightly wrong when a human draws it.
• Rare cases can be requested. If a defect mode occurs once in fifty thousand units, collecting 500 real examples takes years. A renderer produces 500 on demand, which is often the only way to get a class balanced enough to train at all — the general problem in dataset balancing.
• Nuisance factors can be varied independently. You can hold the object fixed and vary only lighting, or hold lighting fixed and vary only pose. Real data never gives you that, and it is what lets you test which factor your model is actually sensitive to.

Notice what is not on that list: raw quantity. A million renders of one CAD model contain roughly one model’s worth of shape information, however many pixels they occupy on disk.

The domain gap, and which half of it matters

A model trained on renders and tested on photographs typically drops, and the drop has two separable causes that call for different fixes.

The first is appearance: renders lack the artefacts a real camera imposes. Sensor noise that rises in shadow, JPEG blocking, chromatic aberration at the frame edge, motion blur, rolling-shutter skew, a slightly dirty lens, the specific colour response of one sensor under one illuminant. A convolutional network is highly sensitive to exactly these low-level statistics, and a clean render is out-of-distribution for a model that will see noisy photographs. This half is cheap to fix, and mostly with augmentation rather than better rendering: add realistic sensor noise, compress to JPEG at the quality your pipeline uses, blur, and vary white balance.

The second is content: what the scenes contain. Renders often have a small pool of backgrounds, physically implausible object placement, and object variety limited to the CAD models somebody had. This half is not fixable by augmentation and is the one that decides whether the exercise works. If your renders show the part on six backgrounds and production shows it on four hundred, no amount of noise simulation closes the gap.

Diffusion-generated images have the mirror-image problem. Their appearance statistics are much closer to photographs, and their labels are not free — you know what you prompted for, not where it ended up in the frame — so they help a classifier more readily than a detector or a segmenter. They also inherit whatever the generator is biased toward, which quietly narrows content diversity in ways that are hard to see by eye.

The mixing ratio, worked on batch composition

There is no universal real-to-synthetic ratio and any page quoting one is guessing. What is universal is the arithmetic that tells you what a chosen ratio does to your small real set, and that is the calculation worth doing before you launch a training run. Assume 400 real images and 20,000 synthetic:

ASSUMPTIONS
  real images                          400
  synthetic images                  20,000
  batch size                            64
  target share of real per batch       25%

per batch:  16 real + 48 synthetic
one pass over the synthetic set = 20,000 / 48 = 417 steps
real images consumed in 417 steps = 417 * 16 = 6,672
                                  = 6,672 / 400 = 16.7 passes over the real set

  -> in the time the model sees each synthetic image once,
     it sees each real image ~17 times

That is the number that decides whether this works. Seventeen passes over 400 images, without strong augmentation, is a recipe for memorising them — and the symptom is confusing, because real validation accuracy climbs early and then falls while synthetic accuracy keeps improving. The levers are all visible in the arithmetic: reduce the real share per batch, augment the real images far more aggressively than the synthetic ones, or freeze most of the backbone so there is less capacity available to memorise with.

The other structure worth knowing is sequential rather than mixed: pretrain on synthetic until convergence, then fine-tune on real alone at a low learning rate. This often beats mixing when the real set is very small, because the final gradient steps come only from the distribution you care about. It costs you the regularising effect of the synthetic data during that final phase, so it wants early stopping on a real validation set. Neither structure is universally better; both are cheap to try, and the sweep over the ratio costs a few GPU hours against months of collecting real images, which is the actual argument for running it rather than reading one.

Domain randomisation instead of realism

The instinct is to make renders more photorealistic. The better-supported strategy is often the opposite, and it has a name. Domain randomisation, introduced by Tobin and colleagues in 2017, varies textures, lighting, camera pose and distractor objects so wildly and so unrealistically that the real world becomes just another sample from the training distribution. The model cannot rely on any particular appearance because none was consistent, so it falls back on the geometry that was.

The argument for the realism route is made by work like Richter and colleagues’ “Playing for Data” (2016), which extracted dense semantic labels from a commercial game engine and showed the resulting data was useful for street-scene segmentation. The two routes are not in conflict so much as suited to different problems: randomisation for a well-defined object whose shape is the signal, realism for scene understanding where context carries the meaning.

One warning about measuring the gap. It is tempting to compare distributions of real and synthetic images with a perceptual distance such as FID and treat a lower number as progress. That measures whether the images look alike to another network, not whether training on them transfers, and the two can move in opposite directions — domain randomisation deliberately increases the distance and improves transfer. The only measurement that answers the question is downstream accuracy on real held-out data.

The rule that cannot be bent

Validate on real data only. Never put a synthetic image in a validation or test split, not even to top up a thin class.

The reason is that the failure this whole exercise risks — a model that has learned render-specific cues — is invisible to a synthetic test set by construction. A model can score 0.98 on held-out renders and 0.61 on photographs, and if your test split is 30% synthetic the reported number sits somewhere between and looks fine. You will discover it in production, which is exactly the validation-to-production gap this technique is most likely to cause.

Two corollaries. Keep a real test set aside from the start, before you render anything, and size it honestly: with 200 real test images a five-point accuracy difference is inside the noise, so a sweep over mixing ratios needs enough test data to resolve the differences it is sweeping. And if real images are so scarce that you cannot spare a test set, the problem you have is not a training-data problem — it is a problem for a few-shot approach, which spends its handful of real images on prototypes rather than on gradient steps.