The Perceptron, and the Book That Stopped It

The story goes that Frank Rosenblatt invented the perceptron, Minsky and Papert proved it could not compute XOR, and neural networks died for seventeen years. Every element of that sentence is either wrong or badly incomplete. The XOR result was old news in 1969, the book’s real theorems are about something else, and the book explicitly says the multilayer case is an open research problem rather than a closed one.

Rosenblatt, 1958, and a machine made of motors

Frank Rosenblatt was a psychologist at the Cornell Aeronautical Laboratory. His paper The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain appeared in Psychological Review in 1958, and his fuller treatment, Principles of Neurodynamics, in 1962. The framing in the title matters: this was presented as a theory of brain organisation that happened to be implementable, not as an engineering technique.

The Mark I Perceptron, built at Cornell around 1958–60, was not a program. It was a physical machine. Input came from a 20-by-20 array of photocells — 400 pixels, and the resolution constraint that implies — wired through a patch panel to association units, whose connections to the input were set randomly rather than learned. The weights that were learned lived in potentiometers, adjusted by electric motors that physically turned the dials during training. Learning was audible.

Structurally, the Mark I is a single layer of trainable weights on top of a fixed random feature map. That description is the key to the whole story, because it is exactly the object Minsky and Papert analysed and it is not what “a neural network” means now.

A perceptron, as analysed in 1969:

  input retina  R  ──►  fixed predicates  φ₁ … φₙ   (not learned)
                              │
                              ▼
                    Σ wᵢ φᵢ(R)  >  θ  ?           (wᵢ learned)
                              │
                              ▼
                          0 or 1

The learning happens only in the weights wᵢ.
Everything interesting about the φᵢ is decided before training.

The press conference and the sentence that followed it

On 7 July 1958 the US Navy demonstrated the perceptron to the press. The New York Times report the following day described the device, on the Navy’s account, as the embryo of an electronic computer that the Navy expected would be able to walk, talk, see, write, reproduce itself and be conscious of its existence.

That sentence is genuine, it is from the newspaper’s report of the Navy’s claims, and it is not a quotation from Rosenblatt’s papers. The distinction is the whole of the hype-versus-source problem in one example. What the machine did in the demonstration was learn to tell cards with a mark on the left from cards with a mark on the right, after fifty trials. What the press release promised was consciousness. Sixty-eight years later, the gap between demonstration and press release is a permanent feature of the field rather than a period detail.

The convergence theorem, and its one condition

The perceptron was not vapour. It came with a genuine theorem, proved in the early 1960s and usually credited to Rosenblatt with an influential independent proof by Novikoff: if a set of training examples is linearly separable in the feature space, the perceptron learning rule will find a separating weight vector in a finite number of updates. Not approximately, not usually — provably, in finitely many steps.

The condition if it is linearly separable is doing all the work, and it is the hinge of everything that followed. The theorem says nothing at all about what happens when the data is not linearly separable, and in that case the rule does not converge; it oscillates indefinitely. So the honest 1962 summary is: a beautiful guarantee, with a precondition nobody knew how to check on real problems.

What Perceptrons actually proves

Marvin Minsky and Seymour Papert published Perceptrons: An Introduction to Computational Geometry with MIT Press in 1969. It is a mathematics book. Its subject is not “can neural networks work” but a precise question: for a predicate to be computable by a perceptron, what does that require of the predicates φ it is built from?

They define the order of a perceptron as the largest number of input points any single φ is allowed to look at, and the diameter as the largest area of the retina any single φ may span. Then they prove results of this form:

• Parity requires order equal to the whole retina. To compute whether the number of active input points is odd, some φ must see every point. There is no way to build parity from local features, at any scale. XOR is the two-input special case of this, and it is the special case because it is easy to draw, not because it is the strong result.
• Connectedness cannot be computed by a diameter-limited perceptron. Whether a figure is one connected blob or two separate ones is not decidable from any collection of features each of which sees only a bounded region. This is the geometrically beautiful result in the book and it is the one that survives.
• Some computable predicates need weights that grow impractically. Even where a perceptron can in principle compute something, the coefficients required may grow exponentially with the size of the retina — a statement about feasibility rather than possibility.

Notice what these results are about. They are about what a fixed, local feature map can support, regardless of how the weights are set. That is a claim about representation, not about learning, and it is not refuted by adding a hidden layer — it is sidestepped by adding a hidden layer, because a learned hidden layer is precisely a way of not having to fix the features in advance. Minsky and Papert knew this perfectly well; it is why the book has a final chapter about multilayer machines at all.

The paragraph everyone cites and nobody quotes

The closing chapter contains the passage the book is actually blamed for. Its structure is worth getting right, because it is more equivocal than its reputation. Minsky and Papert write that the perceptron has proved worthy of study despite — and even because of — its severe limitations, listing its linearity, its learning theorem and its paradigmatic simplicity as attractions. They then say there is “no reason to suppose” that any of these virtues carry over to the many-layered version. And immediately afterwards they describe it as an important research problem to elucidate, or reject, their intuitive judgement that the extension is “sterile”.

Read as written, that is a statement of pessimism plus an explicit invitation to prove them wrong. It is not a proof of impossibility and it does not claim to be one. The 1988 expanded edition added a prologue and epilogue in which the authors addressed the reception directly and disputed the reading that the book had asserted multilayer networks were useless.

What the book did and did not cause

The strong claim — Perceptrons killed neural network research for a generation — is the folklore. Three facts complicate it, and any honest account has to carry all three.

• Funding for connectionist work did decline sharply after 1969, and Minsky and Papert were unusually well placed to influence American funding decisions. Contemporaries in the connectionist camp said at the time that the book was used against them in review panels. That the effect was real is not seriously disputed.
• The research did not stop. Shun-ichi Amari, Kunihiko Fukushima (whose Neocognitron, published in 1980, is a recognisable ancestor of the convolutional network), Teuvo Kohonen, Stephen Grossberg and James Anderson all worked through the 1970s. Paul Werbos’s 1974 thesis contains what we would now call backpropagation applied to neural networks. The winter was in American funding and American prestige, not in the literature.
• The compute was not there anyway. This is the counterfactual worth taking seriously. AlexNet in 2012 needed two GPUs with roughly 1.5 teraflops each for the better part of a week on 1.2 million labelled images. Neither the hardware nor the dataset existed in 1969 at any price, and the silicon that eventually made it possible was funded by an industry that did not exist yet. It is entirely possible that a fully enthusiastic, fully funded connectionist programme in 1970 would have produced roughly what it did produce, because the binding constraint was not belief.

Rosenblatt did not see any of the argument settled. He died in a boating accident in 1971, aged 43, fifteen years before the paper that vindicated the multilayer approach.