Autoencoders and Representation Learning

An autoencoder learns to copy its input through a layer too small to hold it. Nothing else about the setup matters: the narrow layer is the only thing doing any work, and everything autoencoders are used for follows from what a network is forced to discard when it cannot keep everything.

The shapes, and the loss

Two networks and one number:

encoder  f : R^n -> R^k      k < n
decoder  g : R^k -> R^n

loss = || x - g(f(x)) ||^2       mean squared reconstruction error

There are no labels. The target is the input.

A concrete one. A 28 by 28 greyscale image is 784 numbers; encode to 32 and back:

encoder 784 -> 256 -> 32
  784*256 + 256  = 200,960
  256*32  + 32   =   8,224
                   -------
                   209,184 parameters

compression ratio: 784 / 32 = 24.5x

Training is ordinary supervised learning with the labels supplied by the data itself. That is the appeal — and it is why the family matters historically, since it was one of the first arguments that a useful representation could be learned without anyone annotating anything.

What the bottleneck forces

With k smaller than n, the network cannot store the input; it must store a description of it. The 32 numbers have to be enough for the decoder to rebuild 784, which is only possible because the 784 were never independent. Handwritten digits occupy a thin sliver of the space of all 784-pixel images, and the encoder’s job is to find coordinates on that sliver.

Remove the bottleneck and the whole thing collapses. With k >= n and enough capacity, the optimal solution is the identity function, the loss goes to zero, and the code has learned nothing. That failure mode is the reason every interesting variant is really a different way of stopping the network from taking the easy route: shrink the layer, corrupt the input, mask most of it, or penalise the code.

The linear case is PCA

Worth knowing because it anchors the intuition. Make the encoder and decoder single linear layers with no activation and keep the squared error. The optimal solution then spans the same subspace as the top k principal components of the data.

Not the same axes — an autoencoder has no reason to produce orthogonal, variance-ordered directions, and it will happily find any rotation within the subspace — but the same subspace and the same reconstruction error. So a nonlinear autoencoder is exactly “PCA that is allowed to curve”, and the extra capability it buys is following a curved manifold rather than a flat one.

Why a reconstruction code is not an embedding

This is the paragraph most explanations skip, and it decides whether you should reach for an autoencoder at all.

The reconstruction loss asks one thing: can the decoder rebuild the input from the code? Nothing in it asks that similar inputs get similar codes, that the space be smooth, or that distances mean anything. A code that dedicates most of its capacity to whatever varies most in pixel space — brightness, background colour, font weight — scores well, because those are the things reconstruction error punishes.

The embeddings used for retrieval are trained on a different objective entirely: a contrastive loss that pulls related pairs together and pushes unrelated ones apart, so that cosine distance is meaningful by construction. That is why nobody builds a text search index from autoencoder codes, and it is why CLIP is the shape of a modern representation learner and an autoencoder is not.

• Use reconstruction when you want to compress, to denoise, or to measure how unusual an input is.
• Use a contrastive objective when you want distances in the code space to correspond to similarity.
• Use a masked-prediction objective when you want a general-purpose representation to fine-tune from — which is the branch of this family that actually became important.

Three ways to tell whether the code learned anything

A falling reconstruction loss proves only that the decoder is getting better at inverting the encoder, which two networks can achieve while learning nothing you want. Three checks separate a useful code from a well-fitted one, and almost nobody runs them.

1. Beat PCA at the same k. Fit a linear projection to the same code size and compare reconstruction error on held-out data. If the nonlinear autoencoder does not clearly win, the data manifold is close to flat and you have spent a training run to reproduce an eigendecomposition that takes seconds. This is the cheapest baseline in the whole subject and it is skipped almost universally.
2. Interpolate two codes. Take f(x1) and f(x2), decode points along the straight line between them, and look at the results. A plain autoencoder usually produces a crossfade — two overlaid images fading into each other — because nothing ever asked the space between the codes to be occupied by anything. Plausible intermediates mean the space is genuinely continuous, which is exactly the property the variational version is built to guarantee.
3. Fit a linear probe. Train a single linear classifier from the code to a label the autoencoder never saw. If a linear model on 32 dimensions recovers most of the accuracy of a full model on the raw input, the code has organised the data by something meaningful. If it needs a deep probe to extract the label, the information is present but tangled, which for most downstream uses is the same as absent.

The second and third checks are the ones that expose the gap in the previous section. Reconstruction error can be excellent while interpolation is a crossfade and a linear probe is at chance — and that combination is the normal outcome, not a pathology.

The variants that made it useful

Where autoencoders still ship

Inside latent diffusion. The single largest deployment. A convolutional autoencoder compresses a 512 by 512 by 3 image — 786,432 numbers — into a 64 by 64 by 4 latent, or 16,384 numbers. That is 48 times fewer elements, and since diffusion runs dozens of denoising passes over whatever it is given, a 48-fold reduction in the thing being denoised is the difference between affordable and not.

Anomaly detection. Train on normal examples only, then flag inputs whose reconstruction error is high. It works because the decoder has only ever learned to rebuild the normal manifold. The caveat is real and worth stating: a sufficiently general decoder will reconstruct anomalies too, so the method is most reliable when the bottleneck is tight and the normal class is genuinely narrow.

Discrete tokenisers for other modalities. Every model that treats audio or images as token sequences has a vector-quantised autoencoder at the boundary turning continuous signal into indices.

Learned compression. Neural image and video codecs are autoencoders with an entropy model attached to the bottleneck.

The trade

Bought: a representation with no labels, at whatever compression ratio you choose, from a loss that is stable and trivial to monitor.

Cost: the objective optimises for rebuildability, not for meaning. Nothing makes the space smooth, nothing makes the axes interpretable, nothing makes distance correspond to similarity, and under a squared-error loss the model is rewarded for hedging — producing the average of the plausible outputs, which is why plain autoencoder reconstructions look soft. The variational version fixes the first of those and makes the last one worse.