The Technological Singularity, Explained

The word has been through four hands and picked up a different meaning from each. Separating them is most of the work, because the strongest objection to one version is often irrelevant to another.

Where the word comes from

The earliest recorded use in this sense is in Stanislaw Ulam’s 1958 tribute to John von Neumann, reporting a conversation in which the two of them discussed the accelerating pace of technology and the appearance of some essential singularity in human history beyond which affairs as we know them could not continue. It is a passing remark in a memorial article, not a theory. What it establishes is that the intuition predates computing as we know it: it is a claim about the shape of a curve, made by mathematicians looking at technological change generally.

The borrowing is from mathematics and physics, where a singularity is a point at which a function stops being well behaved — where a value diverges or a model stops applying. That metaphor is load-bearing and it is worth noticing which half of it a given writer is using. A divergence claim (things go to infinity) and a breakdown-of-modelling claim (our predictions stop working) are different assertions, and the later authors split along exactly that line.

Good’s ultraintelligent machine

I. J. Good’s 1965 paper Speculations Concerning the First Ultraintelligent Machine contains the argument that everything since has been arguing about. Define an ultraintelligent machine as one that surpasses every human intellectual activity, including machine design. Then, Good argued, such a machine could design better machines; there would be an intelligence explosion; and human intelligence would be left far behind. He added the line that the first such machine would be the last invention humanity need make, provided it is docile enough to tell us how to keep it under control — which is, in one sentence in 1965, both the capability claim and the alignment problem.

Good’s version is a conditional with an explicit premise, which is what makes it worth engaging. It says: if a system exceeds humans at the specific activity of designing intelligent systems, then a feedback loop follows. Whether the loop actually runs depends on premises Good did not defend, and those are the subject of the intelligence explosion argument and its weak points.

Vinge and the horizon of prediction

Vernor Vinge’s 1993 essay The Coming Technological Singularity popularised the term and made a claim different from Good’s. Vinge’s emphasis was epistemic: the creation of superhuman intelligence would be an event past which we cannot model the future, because our models of the future are built by extrapolating from human-level actors. He compared it to a horizon rather than to an explosion, and he listed several routes to it, including intelligence amplification of humans and networked systems, not only a single machine.

That framing has a property people often miss: it is not falsifiable in the usual way, because it is a claim about the limits of prediction rather than about an event. You cannot check it in advance, and after the fact it is nearly unfalsifiable too. This is not a criticism of Vinge, who was clear that he was writing speculation. It is a reason not to treat his version as an empirical forecast.

Kurzweil and accelerating returns

Ray Kurzweil’s version, developed across several books and most fully in The Singularity Is Near (2005), is the one that made the idea popular and it is structurally different again. Kurzweil argues from what he calls the law of accelerating returns: that technological progress is exponential rather than linear because each generation of tools shortens the development of the next, and that this pattern holds across many technologies and long periods. He then extrapolates the curves and reads a date off them.

This is the only one of the four that makes precise, checkable quantitative predictions, and that is a genuine virtue: a specific claim can be wrong, and being wrong in a specific way is informative. It is also where the criticism concentrates, for three reasons that are worth separating. Curve selection: the shape depends on which metrics are plotted and which are omitted, and there is no principled rule fixing the choice. Metric substitution: continuity is often preserved by switching the underlying technology when one plateaus, which is a real phenomenon but makes the composite trend partly a construction of the analyst. And the inference from hardware to capability: a curve about operations per second per dollar does not by itself say what those operations will be able to do, which requires a separate argument about algorithms and data.

Four claims, four evidential standings

Mixing these is the commonest error in discussion of the topic. An objection to curve-fitting says nothing about Good’s conditional. A demonstration that some capability curve is bending says nothing about Vinge’s epistemic claim. When someone says the singularity is nonsense, or that it is obviously coming, the first useful question is which of the four they have in mind.

The objections worth knowing

• Complementary bottlenecks. Even granting rapid cognitive improvement, the physical world imposes serial delays: experiments take time, fabrication takes time, data collection takes time. This is the objection with the most force against fast-takeoff versions specifically, and it is developed further in what recursive self-improvement would actually require.
• Intelligence is not a scalar. François Chollet and others argue that treating intelligence as one quantity that can be multiplied is a category error, and that capability is task-relative and situated. If that is right, “far beyond human” needs restating before it can be evaluated.
• Diminishing returns to cognition. Many hard problems may be limited by information rather than by reasoning — you cannot think your way to a measurement you have not taken. How much this binds depends on the domain, and it binds much less in mathematics and software than in biology or materials.
• Economic history as a base rate. Robin Hanson and others argue that transitions in growth mode have happened before and were fast relative to what preceded them but not instantaneous, and that this reference class is more informative than a mechanism story. Others reply that the reference class contains too few members to support the inference. Both points stand.
• Paul Allen and Mark Greaves’ complexity brake. Their 2011 argument is that scientific progress in understanding intelligence has been slower than exponential because each layer of understanding reveals more complexity, and that software progress does not inherit hardware’s curve.

None of these is a refutation of all four versions at once, and it is worth resisting the impulse to score the topic as won by one side. The honest position after reading the sources is that Good’s conditional is the part with actual content, that its premises are contested by capable people on both sides, and that the popular quantitative version is the weakest link in the chain rather than the strongest.