Generative Adversarial Networks, and Why They Faded

GANs produced the first genuinely convincing synthetic faces and then lost the field to diffusion in about two years. The reasons are structural rather than incidental: the objective is a game rather than a function, nothing in it rewards covering the data, and no number produced during training tells you whether things are going well.

The setup, in shapes

G : R^100 -> image      generator, maps noise to a sample
D : image -> [0,1]      discriminator, probability the input is real

Standard objective:
  min_G max_D  E_x[ log D(x) ] + E_z[ log(1 - D(G(z))) ]

Training alternates:
  1. update D on a batch of real and a batch of fake
  2. update G to make D wrong

A 100-dimensional noise vector goes in, an image comes out, in one forward pass. Note that number: one. It is the property that matters at the end of this page.

The first practical departure from the equation happens immediately. Early in training, D wins easily, so log(1 - D(G(z))) is saturated and its gradient with respect to G is nearly zero — the generator learns fastest exactly when it is already good and not at all when it is bad. Everyone therefore trains G to maximise log D(G(z)) instead. Same fixed point, usable gradients. It is called the non-saturating loss and it is in the original 2014 paper as a footnote to the theory, which tells you something about how much of this field is the difference between the objective and the thing people run.

Why the loss curve tells you nothing

In ordinary training, a falling loss means progress. Here there are two losses and they are in opposition. If D’s loss falls, that may mean D got sharper or that G got worse. If G’s loss falls, that may mean G improved or that D got lazy. At the theoretical optimum D outputs 0.5 everywhere and both losses sit at a constant — success and total failure produce similar-looking curves.

There is no held-out likelihood to evaluate either, because a GAN defines no density. This is why the field ended up with Fréchet Inception Distance — a statistic comparing the distribution of a pretrained classifier’s features on real and generated images — and why in practice GAN training involved a human looking at a grid of samples every few thousand steps. An architecture whose training you cannot monitor automatically is an architecture that does not scale with engineering effort.

Mode collapse is what the objective asks for

Mode collapse is when the generator produces a narrow slice of the data — three digit classes out of ten, one pose, one background. It is usually presented as a mysterious pathology. It is not mysterious at all; it is the objective being satisfied.

Read the generator’s loss again: it rewards fooling D. Nothing in it mentions diversity. If one output fools the current discriminator, emitting that output for every input value of z is an optimal strategy under the loss as written. Diversity is supposed to come indirectly — D notices the repetition and learns to punish it — but that is a dynamic property of the chase, not a term in the objective.

So the failure has a signature. The generator commits to a small set of outputs; the discriminator learns to recognise them; the generator jumps to a different small set; the discriminator follows. The model cycles instead of converging, and the samples look fine in any single grid and are missing most of the data distribution across grids.

Compare with the alternatives. A VAE maximises a bound on the likelihood of every training example, so ignoring part of the data is directly punished — which is why VAEs cover the data and blur it rather than sharpening a subset. Diffusion has the same property for the same reason. Coverage versus sharpness is the fundamental axis here, and GANs sit at one end of it by construction.

Why the training is unstable in principle

Ordinary gradient descent minimises one function, and a function has a descent direction that always exists. Simultaneous gradient descent on two players’ opposed objectives is not minimising anything. The combined update field has a rotational component, which means the natural behaviour is orbiting the equilibrium rather than approaching it — the same way the simplest two-player game with a saddle point cycles forever under gradient dynamics.

That is why the tricks that helped are all attempts to damp the rotation or constrain one player:

Every one of them works. Collectively they are a stack of stabilisers on a process that is not stable underneath, and each adds hyperparameters that interact. This is the practical cost that decided things: a research team could spend a month getting a GAN to converge on a new dataset.

Conditional GANs, and the discriminator that survived

Unconditional generation — noise in, plausible sample out — is the version of the idea people remember, and it is the version that lost. The conditional version had a much longer half-life and is worth separating out.

Conditional GAN: give both players the condition c

  G(z, c) -> sample
  D(x, c) -> is this a REAL x, and does it match c?

c can be a class label, a segmentation map, a sketch, a low-resolution
image, or a photograph in another domain.

Making the discriminator see the condition is the load-bearing part. If it only judged realism, the generator could produce a perfect image that ignores the input entirely. Judging the pair forces correspondence.

• Paired image-to-image. Sketch to photograph, map to aerial view, label mask to scene — the pix2pix formulation (2017), trained on matched pairs with an adversarial loss on top of a plain reconstruction loss.
• Unpaired image-to-image. CycleGAN (2017) removed the need for matched pairs with a cycle-consistency term: translate to the other domain and back, and require that you return to where you started. Two generators, two discriminators, and a constraint that substitutes for supervision.

The most durable piece of engineering from this line is the discriminator design. A PatchGAN discriminator does not emit one verdict for the whole image; it emits a grid of verdicts over overlapping receptive fields, each judging a local region. It has fewer parameters, it applies to any image size, and — the real point — it concentrates the adversarial signal on local texture realism, which is the one thing an adversarial loss is reliably good at.

That is why patch discriminators are still in production today inside things that are not GANs: on the decoder of a latent diffusion autoencoder, in super-resolution, in neural codecs. The architecture faded; this component did not.

What diffusion had that GANs did not

• A regression loss. Add known noise to an image, ask the network to predict the noise, take the mean squared error. It is supervised learning with a target you generated yourself. It goes down when the model gets better, and it goes down at scale.
• Mode coverage by construction. The objective is derived from a likelihood bound over the whole dataset, so dropping part of the distribution costs loss. There is no mode collapse to manage.
• Predictable scaling. More parameters, more data and more compute reliably improved samples, with no per-dataset stabilisation campaign.
• A quality dial at inference. Sampling steps and guidance strength are set after training, so one model serves both a fast preview and a slow high-quality render.

The turn is usually dated to Dhariwal and Nichol’s 2021 paper Diffusion Models Beat GANs on Image Synthesis, and the field moved within about two years. It was not that GAN samples were bad — StyleGAN faces still hold up. It was that diffusion was trainable by anyone.

Where adversarial training is still correct

The honest post-mortem does not end at “GANs lost”, because the one property that made them attractive is one diffusion does not have: a sample costs a single forward pass. A diffusion model needs anywhere from four to fifty network evaluations per sample. Where latency or throughput is the binding constraint, that gap is decisive.

• Distilling diffusion into few steps. Adversarial objectives are used as the distillation loss that turns a many-step sampler into a one- to four-step one. The GAN loss came back as a component rather than as an architecture.
• Neural vocoders. Waveform generation for text-to-speech has run on adversarial models such as HiFi-GAN for years, because audio has to be produced faster than real time and fifty passes per second of audio is not available.
• Decoders inside other models. The perceptual and adversarial loss on a latent diffusion VAE’s decoder is what stops its output being blurry.
• Super-resolution and restoration. Where the transformation is largely determined and only the texture is uncertain, an adversarial loss on top of a reconstruction loss is still a strong and cheap choice.

The trade, stated plainly: GANs bought one-step generation of sharp samples, and paid for it with no likelihood, no usable training signal, no coverage guarantee, and dynamics that do not converge on their own. When compute was scarce that was a bargain. Once compute was abundant, a stable loss was worth more than a fast sampler — and the fast sampler could be recovered afterwards by distillation.