ImageNet and the 2012 Moment

In the 2012 ImageNet competition, AlexNet finished with a top-5 error rate of 15.3 per cent against 26.2 per cent for the runner-up. A ten-point margin in a benchmark where the previous three years had moved a couple of points is what made it a moment. The convolutional architecture it used was published in the 1990s; what did not exist before 2009 was the dataset.

The result, in numbers

The ImageNet Large Scale Visual Recognition Challenge classification task used 1,000 categories, roughly 1.2 million labelled training images, 50,000 validation images and 100,000 test images. The headline metric is top-5 error: the fraction of test images for which the correct label is not among the model’s five highest-scoring guesses.

Read the column downwards and 2012 is not the end of anything; it is the point at which the curve changes shape. The three years after AlexNet moved the error further than the ten years before it, because once one convolutional network won, everyone entered convolutional networks and the field’s entire effort redirected within twelve months.

The dataset nobody wanted to build

ImageNet was Fei-Fei Li’s project, begun at Princeton and continued at Stanford, first presented as a poster at CVPR in 2009 under the title ImageNet: A Large-Scale Hierarchical Image Database. Its structure came from WordNet: each category is a WordNet noun synset, so the labels inherit a hierarchy rather than being a flat list. The full database grew past 14 million labelled images across roughly 21,000 categories; the competition used a 1,000-category subset.

The reason it did not exist earlier is that it was a labour problem rather than a research problem. Labelling millions of images required Amazon Mechanical Turk, which had only launched in 2005, and tens of thousands of workers over a period of years, with quality control layered on top because individual crowd labels are noisy. Li has described the project as difficult to get funded precisely because the computer vision field of the time regarded better algorithms rather than more data as the interesting contribution.

That judgement was the substance of the argument ImageNet settled. The prevailing view was that vision needed better features; the dataset’s implicit claim was that the models were under-determined by the available data and that a large enough labelled corpus would let a high-capacity model find its own features. AlexNet is the demonstration, but the claim is the dataset’s.

What was new in AlexNet, and what was not

The paper is ImageNet Classification with Deep Convolutional Neural Networks, by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, presented at NIPS in December 2012. The network has five convolutional layers and three fully connected layers, about 60 million parameters and 650,000 neurons, ending in a 1,000-way softmax.

Not new: the convolutional architecture itself. Convolution, pooling and weight sharing for images were established by Fukushima’s Neocognitron in 1980 and by LeCun’s work through the late 1980s and 1990s, culminating in LeNet-5, which was reading cheques commercially. Backpropagation was sixteen years old in its Nature form and older than that in the literature.

New, or newly combined at scale:

• ReLU activations instead of tanh. The paper reports reaching a given training error several times faster with rectified linear units than with saturating nonlinearities. This is not a quality improvement so much as a training-time improvement, and training time was the binding constraint.
• Dropout in the fully connected layers. Randomly zeroing half the units during training, to stop the 60 million parameters memorising 1.2 million images.
• Aggressive data augmentation. Random crops, horizontal reflections and colour perturbation — effectively multiplying the training set at no labelling cost.
• A GPU implementation the authors wrote themselves. Two NVIDIA GTX 580 cards, 3 GB of memory each, five to six days of training. The network is split across the two cards in a specific way because 3 GB was not enough for the whole thing — a hardware limit that is visible in the architecture diagram of the published paper, which is as clear an illustration as exists of hardware shaping the model.

So the honest summary of the 2012 result is: a fourteen-year-old architecture, trained with a two-year-old activation function and a one-year-old regularisation trick, on a three-year-old dataset, using gaming hardware. Not one of the ingredients is a conceptual breakthrough. The combination is.

It was not the first GPU network to win a contest

This correction matters because the folklore version credits AlexNet with proving that GPU-trained deep networks could win competitions, and that had already been demonstrated. Dan Ciresan, Ueli Meier and Jürgen Schmidhuber, at IDSIA, won several vision contests in 2011 with GPU-trained convolutional networks — including the IJCNN traffic sign recognition benchmark, where their result was better than the human performance measured on the same task, and Chinese handwriting recognition at ICDAR. Their earlier work had also pushed MNIST error rates down using GPU training.

What AlexNet added was the scale of the problem and therefore the scale of the audience. Traffic signs are a small, constrained domain; 1,000-way natural image classification was the flagship benchmark of a large field, and the margin was impossible to attribute to tuning. The accurate statement is that 2012 was the moment the mainstream computer vision community changed its mind, not the moment the evidence first existed.

The four years that followed

1. 2013. Every serious ILSVRC entry is a convolutional network. Zeiler and Fergus win with an architecture derived from visualising what AlexNet’s layers had actually learned.
2. 2014. Depth becomes the axis. GoogLeNet uses inception modules to go deep without a parameter explosion; VGG goes deep with nothing but stacked 3×3 convolutions and demonstrates that the simple thing works if you can train it.
3. 2015. ResNet’s residual connections make networks of 152 layers trainable by giving the gradient a short path back. This is the architectural idea from this period that survives most visibly — residual connections are in every transformer block in use today.
4. 2015 onwards. Pretrained ImageNet features become the standard starting point for every other vision task. This is arguably the larger consequence: transfer learning from a large pretrained model became routine in vision several years before it became routine in language.

What the benchmark never measured

The “superhuman” claim needs its provenance stated. The human baseline widely quoted for ILSVRC — about 5.1 per cent top-5 error — comes from Andrej Karpathy measuring his own performance on the task in 2014 after training himself on the label set. It is one person, on one benchmark, on a specific 1,000-category vocabulary. It is a useful reference point and it is not a measurement of human visual ability.

Three further limits are worth carrying, because they are exactly the gap between the benchmark and the capability people inferred from it:

• The category distribution is strange. Roughly 120 of the 1,000 classes are dog breeds. A large part of the difficulty — and of the human baseline’s difficulty — is fine-grained discrimination among dogs, which is not what anybody means by image understanding.
• The labels contain errors. Later audits of ImageNet and other standard benchmarks found meaningful rates of incorrect or ambiguous labels in the test sets, which puts a ceiling on what a reported error rate below a few per cent can mean.
• Classification is not perception. Adversarial examples — imperceptible perturbations that flip the label — were demonstrated on these networks within two years, and remain unsolved. A system with 3 per cent top-5 error that can be broken by noise the human eye cannot see has not learned what the benchmark was trying to measure. The same gap between benchmark score and capability is examined generally in why benchmark performance does not transfer.