Skip to content
Multigrid

Architectures beyond the transformer

How the other neural architectures work — convolutions, recurrence, state space models, autoencoders, diffusion, splatting — and what each one traded to get what it got.

The transformer won the part of the field that gets written about, and the effect on everything else has been to make it look settled. It is not. Convolutions still run on every phone. Encoder-decoder stacks still do the speech recognition. Diffusion took image generation away from the architecture that invented it. And the recurrent networks that transformers replaced came back in a form that fixed the reason they were replaced.

These pages explain each of those mechanisms by its shapes: what goes in, what comes out, what a layer actually computes, and how much it costs to run. Where an architecture is presented elsewhere as a successor to attention, the page says what has actually been published and what has not been shown, because the honest answer is usually more interesting than the press release.

Every one of them bought something and paid for it. The trade is stated on each page, in the same place: near the end, in numbers where numbers exist.

Convolutional Networks, and Where They Still Win

What a convolution layer computes, worked through the tensor shapes and the parameter count, and the vision tasks CNNs still own outright.

8 min read

RNNs and LSTMs: What They Solved and What Ended Them

The recurrence, the LSTM gates, the vanishing gradient — and the parallelism loss that actually ended them, which is the less famous of the two answers.

8 min read

State Space Models and Mamba

What a selective state space layer computes, why it trains in parallel and runs with constant memory, and how its state size compares with a transformer's KV cache.

9 min read

RWKV and Recurrent Models With Transformer Quality

How RWKV runs as a strict RNN at inference while training in parallel, what its state actually contains, and the recall cost of a fixed-size memory.

8 min read

Graph Neural Networks

Message passing written as one matrix multiply, the cost of a layer in FLOPs and memory traffic, and one worked use case end to end.

8 min read

Autoencoders and Representation Learning

What the bottleneck actually forces a network to learn, why a reconstruction code is not the same thing as a semantic embedding, and where autoencoders still ship.

8 min read

Variational Autoencoders

The reparameterisation trick explained until it stops being magic, the KL term written out, and the two failure modes that decide whether a VAE works.

9 min read

Generative Adversarial Networks, and Why They Faded

The minimax objective, why mode collapse is what the loss actually asks for, why GAN losses tell you nothing, and where adversarial training is still the right tool.

9 min read

Vision Transformers

How an image becomes 196 tokens, what the patch embedding actually is, and the inductive bias that was traded away — priced in training data.

8 min read

CLIP and Contrastive Multimodal Training

The contrastive loss written as a matrix, why batch size is a capability lever, how zero-shot classification is really retrieval, and what the objective never learns.

8 min read

Encoder-Decoder Models and the Tasks That Still Fit Them

What cross-attention actually does differently, why bidirectional source encoding is a real advantage, and the jobs where the two-stack shape still matches the problem.

8 min read

Pruning, Sparsity, and What the Hardware Can Exploit

Why zeroing half a model's weights usually makes it exactly as slow and exactly as large, which sparsity patterns hardware actually accelerates, and how pruning compares with quantisation.

9 min read

Adapters, Hypernetworks and Modular Weights

The family LoRA belongs to, the structural reason it won — parallel rather than sequential — and the members that trade differently: prefix tuning, IA3, BitFit, hypernetworks.

9 min read

Retrieval Built Into the Architecture

What kNN-LM, RETRO and memory layers do differently from prompt-level RAG, why retrieved text can avoid the context window entirely, and why the prompt-level version won anyway.

9 min read

Diffusion Language Models

How denoising a whole sequence in parallel replaces one-token-at-a-time decoding, the latency argument as arithmetic, and where the argument stops holding.

9 min read

Energy-Based Models and Prediction in Representation Space

What an energy function is, why the normalising constant is the whole difficulty, and the JEPA argument for predicting representations rather than pixels.

9 min read

Neural Radiance Fields and Gaussian Splatting

What it means for a scene to become weights, why rendering a NeRF frame costs a hundred million network evaluations, and what splatting changed by deleting the network.

9 min read

Spiking Neural Networks and Neuromorphic Hardware

What a spiking neuron computes, why the energy argument is real at the circuit level, why the gradient does not exist, and an honest account of where the hardware actually is.

9 min read

Byte-Level and Tokenizer-Free Models

What disappears when tokenisation does — the vocabulary, the language tax, the spelling failures — and what it costs in sequence length, priced.

9 min read

Hybrid Architectures: Attention Plus Something Else

Why new models interleave layer types instead of choosing one: the KV-cache arithmetic that forces it, the layer types being mixed, and how to read a model card for the mix.

9 min read

Gaussian Splatting, Explained

What one 3D Gaussian holds as data, why 59 floats per splat becomes hundreds of megabytes per scene, and why projecting and sorting beats marching rays.

11 min read

Other topics