The Turing Test: What Turing Wrote, and What People Think He Wrote

Alan Turing’s Computing Machinery and Intelligence appeared in Mind in October 1950. It proposes replacing the question “can machines think?” with a game, and it makes one dated prediction about how well machines would play that game by the year 2000. It does not define a pass mark, it does not call the game a test, and Turing never called it the Turing test. Almost everything the phrase now means was added by other people.

The paper, and what it replaces

The paper opens by stating that Turing proposes to consider the question “Can machines think?” — and then immediately argues that answering it would require definitions of machine and think drawn from ordinary usage, which would reduce the question to a survey of what people happen to mean by the words. He calls that absurd, and replaces the question with a different one that he considers closely related and expressible in relatively unambiguous words.

That substitution is the whole architectural move of the paper, and it is the part most summaries drop. Turing is not offering a definition of thinking. He is arguing that the original question is bad and proposing an operational replacement, on the grounds that the replacement can actually be run. Later in the paper he says plainly that he believes the original question to be too meaningless to deserve discussion.

The imitation game, as originally described

The original description is not two players and a judge deciding human-or-machine. It is three people:

1. A man, labelled A.
2. A woman, labelled B.
3. An interrogator, C, of either sex, in a separate room, who communicates with both by teleprinter so that voice and handwriting give nothing away.

The interrogator’s task is to determine which of A and B is the man and which is the woman. A’s object is to cause the interrogator to make the wrong identification; B’s object is to help the interrogator. Only then does Turing ask the question that matters: what happens when a machine takes the part of A in this game? Will the interrogator decide wrongly as often as when the game is played between a man and a woman?

Scholars have argued for decades about whether the gender framing is load-bearing or merely a device for introducing the idea of judging by text alone. The reading that the machine must imitate a woman while competing against a woman gives a genuinely different test from the standard modern reading, in which a machine competes against a human and the judge guesses which is which. Turing’s own later restatements in the paper, and in a 1952 BBC radio discussion, are closer to the modern reading — the jury has to decide which of two respondents is a machine — which suggests he did not consider the gender detail essential. It is worth knowing that the ambiguity is in the source rather than pretending the source is clean.

The one number, and what it was a claim about

The famous percentage comes from a single sentence in section 6, and the sentence is a forecast about engineering progress, not a criterion for intelligence. Turing wrote that he believed that in about fifty years’ time it would be possible to programme computers with a storage capacity of about 10⁹ to make them play the imitation game so well that an average interrogator would not have more than a 70 per cent chance of making the right identification after five minutes of questioning.

Unpack the three quantities, because each one is routinely mangled:

The second half of that same passage is the prediction Turing appears to have cared about, and it was about language rather than machines: he expected that by the end of the century general educated opinion would have altered so much that one would be able to speak of machines thinking without expecting to be contradicted. That is a claim about usage, and on that one he was broadly right.

The nine objections he answered

Over half the paper is a section titled “Contrary Views on the Main Question”, in which Turing states and answers nine objections. Listing them is the fastest way to show how much of the modern debate he had already anticipated in 1950.

• The theological objection. Thinking requires a soul, which God gives only to humans.
• The “heads in the sand” objection. The consequences would be too dreadful, so let us hope it is impossible. Turing treats this one dismissively and does not pretend otherwise.
• The mathematical objection. Gödel and the halting problem impose limits on machines. Turing’s answer — that no proof has been offered that humans are not subject to comparable limits — is the same answer given to Lucas in 1961 and to Penrose in 1989.
• The argument from consciousness. Stated via Professor Geoffrey Jefferson’s 1949 Lister Oration. Turing replies that the position leads to solipsism: the only way to be sure a machine thinks is to be the machine.
• Arguments from various disabilities. A machine can never do X, for a long list of X: enjoy strawberries and cream, make mistakes, be the subject of its own thought, learn from experience.
• Lady Lovelace’s objection. Ada Lovelace’s 1843 note that the Analytical Engine has no pretensions to originate anything. Turing’s reply is that machines frequently surprise him.
• Argument from continuity in the nervous system. The brain is not discrete-state, so a discrete-state machine cannot imitate it.
• The argument from informality of behaviour. Humans do not follow rules, so no set of rules can capture them. This is the objection Dreyfus later built a career on.
• The argument from extra-sensory perception. Turing takes telepathy seriously enough to devote a passage to it and to suggest a telepathy-proof room. This is the section every summary omits, and it is a useful reminder that a document is a product of its year even when it is a work of genius.

The paper also proposes something that reads very oddly in 1950 and very familiarly now: rather than trying to programme an adult mind, programme something with the capacity of a child’s and then educate it. Turing spends the last pages on learning machines, including the suggestion that random elements in the learning process are useful. The end of the most cited paper in AI is an argument for learning over hand-coding.

Five things the paper does not say

1. It never uses the phrase “Turing test”. The name is later, applied by other authors.
2. It sets no pass mark. The 30 per cent figure is the complement of a forecast about the year 2000. Treating it as a threshold for machine intelligence is reading a weather forecast as a legal standard.
3. It does not claim the game measures intelligence. Turing offers it as a workable substitute question, having argued that the original question is not worth answering.
4. It does not require deception about being a machine. The imitation is of a human respondent in a game both parties know they are playing, not a con.
5. It does not say a machine that fails is not thinking. Turing explicitly notes the asymmetry — the game is a sufficient demonstration if passed, not a necessary condition.

What people have done with it since

The Loebner Prize ran an annual imitation-game contest from 1990, funded by Hugh Loebner, through most of the following three decades. It was widely criticised by researchers, including for rewarding conversational trickery over anything resembling understanding, and it wound down at the end of the 2010s. The entrants that won it were typically pattern-matching systems in the direct line of descent from ELIZA, not attempts at general capability.

The 2014 “pass” is the case worth understanding, because it shows how the folklore version of the test behaves in practice. At a Royal Society event marking the sixtieth anniversary of Turing’s death, a program called Eugene Goostman was reported to have convinced 33 per cent of judges in five-minute conversations. The program presented itself as a thirteen-year-old boy from Odessa speaking English as a second language — a persona chosen so that non-sequiturs, patchy grammar and gaps in general knowledge read as characterful rather than as failure. It cleared the folklore threshold exactly, and told nobody anything about machine intelligence.

The genuinely interesting modern question is the opposite of the one Turing posed. Systems now produce text that most people cannot distinguish from human writing in short exchanges, and this turns out to say much less about their capabilities than the 1950 framing assumed — fluency and competence came apart in a way the paper does not anticipate. If you want the mechanism behind that gap, it is the subject of why fluent output and correct output are different achievements.