Backpropagation's Long Road to Acceptance

Backpropagation was independently discovered at least half a dozen times between 1960 and 1986, in control theory, in numerical analysis and in neural network research, mostly by people who had not read each other. The question “who invented it” has no single answer because it is three questions wearing one coat.

The question has three answers

Separate these and the history stops being contentious:

Most of the heat in the credit dispute comes from conflating the first and third rows. Almost nobody claims the 1986 paper was first in the strict sense — including its authors, who have said so repeatedly in the decades since.

What the algorithm is

Backpropagation is the chain rule applied to a composed function, in reverse order, with intermediate results reused. That is the entire content of it, and the reason it was rediscovered so many times is that anyone who writes the chain rule down for a deep composition and then notices they are recomputing the same partial products will arrive at it.

The property that makes it an algorithm rather than an observation is locality. Each node in the computation needs to know only two things: how its own output depends on its own inputs, and the gradient of the loss with respect to its output, which the node above hands down. It multiplies the two and passes the result to the nodes below. No node needs any knowledge of the network’s overall shape.

A two-layer network, written out completely.

  forward:   z₁ = W₁x + b₁
             a₁ = σ(z₁)
             z₂ = W₂a₁ + b₂
             L  = loss(z₂, y)

  backward, each line using only the line above it:
             δ₂ = ∂L/∂z₂                       (from the loss)
             ∂L/∂W₂ = δ₂ a₁ᵀ                   (local: z₂ = W₂a₁ + b₂)
             ∂L/∂b₂ = δ₂
             δ₁ = (W₂ᵀ δ₂) ⊙ σ'(z₁)            (local: chain through σ)
             ∂L/∂W₁ = δ₁ xᵀ
             ∂L/∂b₁ = δ₁

Every gradient in the network comes out of that pattern repeated.
Add a hundred more layers and nothing changes except the number of
times the middle two lines run.

Note where δ₁ comes from: it is δ₂ pushed backwards through W₂ and
then multiplied by σ'(z₁). That multiplication is the whole of the
vanishing gradient problem, and it is discussed below.

Because the rule is local, an implementation does not need to know what network it is differentiating — which is why modern frameworks can differentiate arbitrary code rather than a fixed architecture, and why writing a new architecture does not require deriving anything by hand. That generality is Linnainmaa’s contribution rather than the neural network community’s, and it is the reason the first row of the table above is a separate question.

The discoveries, in order

• 1960 — Henry J. Kelley, in Gradient theory of optimal flight paths, derived a continuous-time version of the backward gradient computation for multi-stage trajectory optimisation. Arthur Bryson published closely related work shortly afterwards. This is control theory and has nothing to do with neurons; the mathematics is the same.
• 1962 — Stuart Dreyfus published a derivation of the multistage gradient using the chain rule directly, a considerable simplification of the control-theoretic treatments.
• 1960s — Shun-ichi Amari in Japan worked on training multilayer networks by stochastic gradient descent. His contribution is well attested and was largely unread in the West at the time, which is a recurring feature of this timeline.
• 1970 — Seppo Linnainmaa, in a master’s thesis at Helsinki, gave the general method for the cumulative rounding error of an algorithm as a composition of elementary operations — which is reverse-mode automatic differentiation over an arbitrary computational graph, with code. This is the earliest publication of the general algorithm in the form used today.
• 1974 — Paul Werbos, in Beyond Regression (Harvard PhD thesis), proposed applying reverse-mode differentiation to train neural networks. This is the first statement of backpropagation-for-neural-networks as such. It attracted very little attention, in a period when neural networks were nearly unfundable in the United States.
• 1985 — David Parker and, independently, Yann LeCun — LeCun’s in French, at a francophone conference — published derivations again, still independently.
• October 1986 — Rumelhart, Hinton and Williams, Learning representations by back-propagating errors, Nature. The paper that made it standard.

The pattern is that the algorithm is a straightforward consequence of the chain rule applied to a composed function, so anyone who needs it badly enough will derive it. What was scarce was not the derivation but a reason to want it, and that reason only became compelling when somebody showed what the hidden layers did with it.

Why reverse mode, and the arithmetic that decides it

There is a competing way to get a gradient, and comparing the two explains why backpropagation is not merely convenient but necessary. The comparison is arithmetic and it does not go stale.

Two ways to get the gradient of one loss with respect to N parameters.

FINITE DIFFERENCES
  For each parameter i, nudge it by ε and re-evaluate the loss:
      ∂L/∂wᵢ  ≈  ( L(w + ε·eᵢ) − L(w) ) / ε
  Cost: one forward pass per parameter.  →  N forward passes.

REVERSE-MODE (BACKPROPAGATION)
  One forward pass storing intermediate activations, then one
  backward pass applying the chain rule to every node in reverse
  topological order.
  Cost: roughly 2× one forward pass, INDEPENDENT of N.

WORKED EXAMPLE — a small 1986-era network
  N = 10,000 parameters, forward pass = 1 ms
      finite differences : 10,000 × 1 ms   = 10 seconds per gradient
      backpropagation    : ~2 × 1 ms       = 2 milliseconds
      ratio              : 5,000×

WORKED EXAMPLE — a 7-billion-parameter model
  N = 7 × 10⁹, forward pass = 100 ms
      finite differences : 7×10⁹ × 100 ms  ≈ 22 years per gradient step
      backpropagation    : ~200 ms
      ratio              : ~3.5 × 10⁹

ASSUMPTIONS: the forward pass cost is taken as constant; finite
differences is one-sided (central differences doubles it); the
backward pass is taken as twice the forward, which is the usual
rule of thumb and is accurate to within a small factor.

The ratio is the parameter count. That is why every large model is trained by reverse-mode differentiation and why no alternative has ever been competitive: the cost of the gradient is decoupled from the number of things you are differentiating with respect to. Everything in modern deep learning rests on that one property, and the price paid for it is memory — you must keep the forward activations to run the backward pass, which is why activation checkpointing exists and why training memory exceeds inference memory by so much.

Why 1986 is the date everyone remembers

The Nature paper is short and its contribution is not the algorithm. Its contribution is the demonstration of what the algorithm produces: the paper shows that the procedure causes hidden units to develop internal representations — features that are not present in the input encoding and that the network invented because they were useful for the task.

That is the direct answer to the objection that had held the field back. The 1969 critique in Minsky and Papert’s Perceptrons was, at bottom, that a perceptron’s features are fixed in advance, so its capabilities are bounded by whatever the feature designer thought of. If the features can be learned, that bound disappears. The 1986 paper is the existence proof, in four pages, with worked examples including symmetry detection and a family-tree relation task.

Three other things made 1986 the moment rather than 1974. The Parallel Distributed Processing volumes appeared the same year and gave the connectionist programme a manifesto and a textbook. Hardware had improved to the point where small networks could be trained in minutes on a workstation rather than in hours on a shared mainframe. And the symbolic programme was, by 1986, visibly running into the maintenance wall described on the expert systems page, so there was an audience for an alternative.

Why it still did not work for twenty years

Here is the part of the story that the credit argument obscures. Having backpropagation in 1986 did not give anybody deep networks. Networks of more than two or three hidden layers were essentially untrainable for the next two decades, and the reason was identified early.

Look again at the backward line for δ₁ in the derivation above. Each layer the gradient passes through multiplies it by a weight matrix and by the derivative of the activation function. The classical activations — the logistic sigmoid and tanh — have derivatives bounded well below one over most of their range, and near zero once a unit saturates.

Gradient magnitude after passing back through L layers

  Assume each layer multiplies the gradient by an average factor k.

      k = 0.5,  L = 10  →  0.5¹⁰  ≈ 0.001
      k = 0.5,  L = 30  →  0.5³⁰  ≈ 9 × 10⁻¹⁰
      k = 1.5,  L = 30  →  1.5³⁰  ≈ 1.9 × 10⁵

Below one, the gradient reaching the early layers is numerically
negligible and those layers never learn. Above one, it explodes and
training diverges. The stable band is narrow and nothing in the
algorithm keeps you inside it.

ASSUMPTION: a single average factor per layer, which is a caricature —
the real analysis is about the spectrum of the Jacobian. The
exponential dependence on depth is the part that survives the
caricature.

Sepp Hochreiter analysed this in his 1991 diploma thesis, and Yoshua Bengio and colleagues published on the difficulty of learning long-term dependencies with gradient descent in 1994. The vanishing gradient is not a bug in backpropagation; it is a property of differentiating a deep composition, and it explains why the 1986 result produced excitement, then shallow networks, then the second AI winter.

The fixes arrived separately and none of them changed the algorithm. LSTM, from Hochreiter and Schmidhuber in 1997, gave recurrent networks a path along which the gradient is not repeatedly multiplied. Layer-wise unsupervised pretraining, around 2006, initialised deep networks somewhere the gradient could still be useful. Rectified linear units, whose derivative is exactly one wherever the unit is active, removed the saturation for feed-forward networks. Careful initialisation schemes set the initial factor near one deliberately. Batch and layer normalisation kept it there during training. And residual connections, from 2015, gave the gradient a direct path around every block, which is why 152-layer networks became trainable and why every transformer block has one.

So the honest account of the twenty-six years between the Nature paper and AlexNet is not that the field ignored a working method. The method worked and the objects it was applied to did not train, and closing that gap took a decade of separate discoveries, none of which is called backpropagation.

The credit argument, and what is actually contested

From around 2015 Jürgen Schmidhuber published a series of detailed credit-assignment articles arguing that the standard history systematically under-cites Linnainmaa, Werbos, Amari and others, and that the field’s citation practices reward the popularisers rather than the originators. The 2018 Turing Award, given to Yoshua Bengio, Geoffrey Hinton and Yann LeCun, sharpened the dispute considerably.

What is not contested, on any side:

• Linnainmaa published the general algorithm in 1970 and Werbos applied it to neural networks in 1974.
• The 1986 Nature paper does not claim priority for the algorithm, and its authors have publicly credited Werbos and others many times since.
• The 1986 paper is nonetheless the reason the technique became standard practice, and its contribution — the internal representations result — is a genuine and separate one.

What is contested is a question about norms rather than facts: whether the field’s citation conventions should track first publication or effective transmission. It is a real disagreement and it is not settled by knowing the dates. The dates, at least, are not in doubt, and having them in order is the part a reader can use.