RNNs and LSTMs: What They Solved and What Ended Them

The standard answer is that recurrent networks died of vanishing gradients. That is half right and chronologically wrong: the LSTM largely fixed vanishing gradients in 1997 and recurrent models stayed state of the art for another twenty years. What ended them was that a recurrent layer cannot be computed in parallel across time, and by 2017 parallelism was the only currency that mattered.

The recurrence, in shapes

A recurrent layer carries a hidden state vector h of size d from one timestep to the next. At each step it reads one input vector and produces one new state:

h_t = tanh(W_xh @ x_t + W_hh @ h_{t-1} + b)

x_t : (n,)      the input at this step
h_t : (d,)      the state, carried forward
W_xh: (d, n)    input projection
W_hh: (d, d)    the recurrent matrix, applied once per step

The parameter count is independent of sequence length, which was the original selling point. With n = d = 512: 512*512 + 512*512 + 512 = 524,800 parameters, whether the sequence is ten tokens or ten thousand.

The state is the only channel between the past and the present. Every token the model has seen must be compressed into those d numbers, and the compression is lossy in a way nothing can undo later. Hold on to that, because it is the same constraint that state space models live under today.

What the LSTM cell adds

The LSTM splits the state in two. Alongside the hidden state h it carries a cell state c, and it adds four small gate networks that decide what happens to it:

f_t = sigmoid(W_f @ [x_t, h_{t-1}])     forget gate
i_t = sigmoid(W_i @ [x_t, h_{t-1}])     input gate
g_t = tanh   (W_g @ [x_t, h_{t-1}])     candidate
o_t = sigmoid(W_o @ [x_t, h_{t-1}])     output gate

c_t = f_t * c_{t-1} + i_t * g_t         the cell update
h_t = o_t * tanh(c_t)

Four gates means four times the parameters of a vanilla RNN: 4 * (n*d + d*d + d), which at n = d = 512 is 2,099,200.

The load-bearing line is c_t = f_t * c_{t-1} + .... The old cell state is multiplied by a gate and added to, not passed through a matrix and a squashing nonlinearity. When the forget gate sits near 1, information travels from step to step essentially untouched. That additive path is the LSTM’s entire contribution, and it is why the architecture held on for two decades.

The vanishing gradient, in numbers

To train on a sequence of length T, the gradient at step T has to reach step t, and it does so through a product of T - t Jacobians. A product of matrices behaves like a product of scalars in the dominant direction:

spectral radius 0.9, 100 steps:  0.9^100  = 0.0000266
spectral radius 1.1, 100 steps:  1.1^100  = 13,780
spectral radius 1.0, 100 steps:  1.0^100  = 1

Below one and the gradient disappears; above one and it explodes; the knife edge is exactly one and nothing keeps it there. Exploding gradients have a crude and effective fix — clip the norm — and vanishing ones do not, because there is nothing to clip. A signal that arrived as 2.66 times ten to the minus five is not distinguishable from noise.

In an LSTM the corresponding derivative along the cell path is f_t, the forget gate. With a gate near 1 the product stays near 1 over hundreds of steps. That is the fix, it works, and it was published in 1997.

The loss that actually ended them

Look again at h_t = f(h_{t-1}, x_t). Step t cannot begin until step t-1 has finished. Not at inference — that is true of any autoregressive model — but at training time, which is where the constraint does its damage.

Put numbers on it. Sequence length 2,048, batch 64, hidden size 1,024:

RNN training pass, one layer:
  2,048 sequential steps
  each step: (64 x 1024) @ (1024 x 1024) = 67 million MACs
  2,048 separate kernel launches, each dependent on the last

Transformer training pass, one layer:
  1 step
  all positions at once: (64*2048 x 1024) @ (1024 x 1024)
                       = 137 billion MACs in one matmul

The total arithmetic is comparable. The shape of it is not. A 67-million-MAC matmul does not come close to saturating a modern accelerator, so each of those 2,048 steps is dominated by launch overhead and memory latency rather than by arithmetic, and the machine idles through most of the pass. The transformer issues one enormous matmul that the hardware is designed for.

Worse, the constraint does not yield to more hardware. Adding GPUs lets you process more sequences at once, but the wall-clock time for one sequence is still 2,048 dependent steps. The time axis is the one dimension of the problem that cannot be sharded.

Which one mattered

Both are real; only one was decisive. The vanishing gradient was a correctness problem and it had a solution that worked. The parallelism loss was an economics problem and it had none.

Between roughly 2012 and 2017 the winning move in machine learning became “train a larger model on more data using more accelerators”. Every architecture that could convert a bigger budget into a better model got better; every architecture that could not, stopped. Recurrence could not, because its budget was spent on sequential latency rather than on arithmetic. The 2017 transformer paper is called Attention Is All You Need, but the abstract leads with the training time, and that is the honest ordering.

Where attention came from

The history is worth two paragraphs because it says what attention actually is.

Sequence-to-sequence translation (Sutskever and colleagues, 2014) ran an LSTM over the source sentence, took the final hidden state as a summary of the whole thing, and decoded the translation from it. One fixed-size vector for a sentence of any length. Quality degraded as sentences got longer, which is exactly what the shape predicts: the state is a bottleneck of constant width and a longer sentence has more to push through it.

Bahdanau, Cho and Bengio (2014) proposed the fix. Instead of decoding from the final state alone, let the decoder compute, at each output step, a weighted sum over all the encoder’s hidden states, with the weights produced by a small learned scoring network. The recurrence stayed; a way to look back at everything was added beside it. That is attention, invented as a patch to a recurrent model.

Three years later the transformer removed the recurrence and kept the patch. Which tells you what the two mechanisms are for. The recurrent state is compression: fold the past into a fixed width. Attention is refusal to compress: keep everything and decide later what to look at. That is the single axis this whole subject moves along, and the modern recurrent architectures are travelling back down it deliberately, having decided the memory cost of refusing to compress is too high.

One footnote on the family. The GRU merges the forget and input gates into a single update gate and drops the separate cell state, giving three gate computations instead of four and about three-quarters of the parameters. On most tasks the difference from an LSTM is inside the noise, which is itself informative: the specific gate arrangement mattered much less than having an additive path at all.

What survives

• The gating idea. Multiplicative gates that decide what to keep are everywhere now — in selective state space models, in RWKV, in the gated MLPs inside current transformer blocks.
• Small streaming models. Keyword spotting, on-device wake words and some streaming speech front ends still use GRUs and LSTMs, because the model is small enough that parallelism never mattered and constant per-step cost does.
• The problem statement. “Fixed-size state, constant cost per token, parallelisable training” is a coherent specification, and the architectures in the next two pages are attempts to satisfy all three at once rather than two of the three.