The Hardware Accident That Made Deep Learning Possible

The chips that train and serve every large model descend from hardware built to draw shaded triangles at sixty frames per second, and the research and development that made a chip with thousands of arithmetic units affordable was paid for by people buying graphics cards to play games. That is the accident. The reason it worked is not an accident at all, and it is two numbers.

The chronology, 1993 to 2012

1. Early 1990s. Real-time 3D games create a consumer market for rasterisation hardware. Dedicated accelerator cards arrive mid-decade and sell in volumes that fund serious silicon development.
2. 1999. NVIDIA releases the GeForce 256 and markets it as the first “GPU”, on the basis that it performs transform and lighting on the chip rather than on the CPU. The pipeline is fixed-function: it does what it does and you configure it.
3. 2001. Programmable shaders arrive with DirectX 8 and the corresponding hardware. Game developers can now write small programs that run per vertex and per pixel. This is the pivotal change and it was made entirely for graphics reasons.
4. 2002–2006. The GPGPU period. Researchers who want the arithmetic throughput express their computations as graphics operations — data goes in as textures, the computation is a pixel shader, results come back as a rendered image. It works and it is miserable, and a handful of academic languages appear to make it less so.
5. 2006–2007. NVIDIA ships a unified shader architecture and, with it, CUDA — a C-like language for running arbitrary parallel code on the GPU with no graphics involved. This is the decision that mattered commercially, and it was taken before there was a market to justify it.
6. 2005–2010. Neural networks on GPUs appear in the literature. A 2009 paper from Stanford on large-scale unsupervised learning with graphics processors reports speed-ups of roughly seventy times over CPU implementations for deep belief networks. Work at IDSIA around 2010 reports comparable factors for convolutional networks and pushes handwritten digit error rates to new lows.
7. 2011. GPU-trained convolutional networks win several vision competitions, including a traffic sign benchmark on which the reported result exceeded the human performance measured on the same task.
8. 2012. AlexNet wins ImageNet on two GTX 580 cards with 3 GB of memory each, after five to six days of training. The network is split across the two cards specifically because the model does not fit in 3 GB — a consumer memory limit visible in the architecture of the most cited vision paper of the decade.

The accident: shaders made it programmable

The single contingent event in that chronology is 2001. A fixed graphics pipeline is useless for anything but graphics; a programmable one is a parallel computer. Programmable shaders were introduced because game developers wanted lighting and material effects the fixed pipeline could not express, which is a demand with nothing whatever to do with linear algebra.

What that demand bought, as a side effect, was a chip designed around a very specific set of assumptions: run the same short program over many data elements at once; tolerate high memory latency by having enormous numbers of threads in flight; spend almost all the transistor budget on arithmetic units rather than on caches, branch prediction and out-of-order execution. Those assumptions were chosen because they suit shading a million pixels. They also happen to be exactly the assumptions that suit multiplying large matrices, and nobody in 2001 chose them for that reason.

Why matrix multiplication fits, derived

The two numbers that decide whether any computation suits this hardware are its arithmetic throughput and its memory bandwidth. The ratio between them gives a threshold, and comparing a computation’s own ratio against that threshold tells you which resource you are limited by. This is the roofline argument and it is worth doing explicitly, because it explains both why training loves GPUs and why inference behaves so differently.

ARITHMETIC INTENSITY = floating-point operations ÷ bytes moved

Step 1 — the machine's threshold.
  Take a modern datacentre accelerator, approximately:
      dense BF16 throughput   ≈ 1,000 × 10¹² FLOP/s
      HBM memory bandwidth    ≈ 3.3 × 10¹² bytes/s
  threshold = 1,000e12 / 3.3e12  ≈ 300 FLOPs per byte moved

  Below 300 FLOPs/byte you are memory bound: the arithmetic units
  wait for data. Above it you are compute bound.

Step 2 — a square matrix multiply, N × N by N × N.
      FLOPs  = 2N³
      bytes  = 3N² × 2   (three matrices, 2 bytes per BF16 value)
      intensity = 2N³ / 6N² = N/3 FLOPs per byte

      N = 1,024   →  341 FLOPs/byte   — just compute bound
      N = 4,096   →  1,365            — comfortably compute bound
      N = 16,384  →  5,461            — the machine is fully fed

  Intensity grows linearly with N. Big matrix multiplication is the
  one common operation that gets MORE suited to this hardware as it
  gets larger, which is why model dimensions are what they are.

Step 3 — the same numbers on a CPU-shaped machine.
  A CPU spends its transistors on caches, branch prediction and
  out-of-order execution, all of which accelerate irregular,
  dependent, branchy code. A matrix multiply is none of those things:
  it is regular, independent and branch-free. Every transistor spent
  on those features is a transistor not spent on arithmetic.

ASSUMPTIONS: peak specification figures, dense rather than sparse,
no cache reuse modelled, and the 3N² memory term assumes each matrix
is read or written exactly once. Real kernels do better through
tiling. The specification numbers move with each hardware generation;
the shape of the argument does not.

Why generating one token does not fit

Run the same calculation for the operation a language model performs when it generates a single token for a single request, and the answer inverts. This is the most useful thing on this page for anyone operating a model rather than reading about one.

GENERATING ONE TOKEN, BATCH SIZE 1

  The operation is matrix-VECTOR, not matrix-matrix: one token's
  activation vector against the full weight matrices.

      FLOPs  ≈ 2 × (number of parameters)
      bytes  ≈ 2 × (number of parameters)     [BF16, read once]
      intensity ≈ 1 FLOP per byte

  Against a threshold of ~300 FLOPs/byte, that is memory bound by a
  factor of roughly 300. The arithmetic units are idle almost all of
  the time; the chip is a very expensive way to stream weights out of
  memory.

WHAT BATCHING DOES

  Process B requests together and the same weight read serves all B:
      FLOPs  ≈ 2 × B × params
      bytes  ≈ 2 × params        (weights read once for the batch)
      intensity ≈ B FLOPs per byte

  So the batch size IS the arithmetic intensity, and you need a batch
  in the hundreds to reach the compute-bound regime.

CONSEQUENCE, in one line: a served model wants a large batch and a
single interactive user cannot supply one. That is why per-token
prices are what they are, and why a local model on one machine gets a
tiny fraction of the hardware's rated throughput no matter what the
specification sheet says.

ASSUMPTIONS: dense model, BF16 weights, attention KV-cache traffic
excluded (it makes the memory-bound case worse at long context, not
better), and one forward pass per token.

That derivation is the whole economics of model serving in twenty lines, and it is why the mechanics of training compute and inference compute behave so differently despite running on the same silicon.

When the accident stopped being one

The hardware has been purpose-built for this workload for about a decade now, and the graphics heritage is increasingly vestigial. The decisive change was tensor cores — arithmetic units that perform a small matrix multiply and accumulate as a single instruction — introduced in the datacentre line in 2017 and present in every generation since. A tensor core is of no use whatever for drawing triangles. It exists because of deep learning.

Since then: successive generations have added memory bandwidth, faster inter-chip interconnect for splitting a model across many devices, and progressively narrower numeric formats — 16-bit, then 8-bit, and lower — each of which halves the bytes moved per parameter and therefore doubles the effective arithmetic intensity of a memory-bound operation. Read the derivation above and every one of those changes is aimed at exactly the bottleneck it identifies.

Google’s tensor processing units went the other way from the start — designed for neural network arithmetic, with no graphics lineage at all, deployed internally from around 2015 and described publicly in a 2017 architecture paper. Their existence is the proof that the graphics route was contingent rather than necessary: given the workload, you can design the chip directly, and the reason nobody did so earlier is that the workload was not worth a chip.

What the accident story leaves out

“Video games funded deep learning” is true and it is not the whole story, and the parts it omits are the parts that were decisions rather than luck.

• CUDA was a bet, not a windfall. Building a general-purpose programming model and supporting it across every product line, from 2006 onwards, cost real money at a time when the scientific computing market was small and the machine learning market did not exist. The software ecosystem that resulted — the libraries, the compiler, the years of accumulated kernels — is a larger barrier to competitors today than the silicon is.
• Researchers had to do the porting. The 2005–2012 results in the chronology above required people to write neural network code in a graphics-derived programming model for years before anyone had a framework. AlexNet’s training code was written by its authors, not called from a library.
• The economics inverted, and the record is public. Gaming was the dominant source of revenue for the leading vendor for most of its history; datacentre revenue overtook it in its fiscal year 2023 and has since dwarfed it. The consumer market that subsidised the silicon is now the smaller line item, which is as clean a marker as exists for when the accident became the business.

The transferable observation is about how enabling technologies actually arrive. The hardware that made modern machine learning possible was not developed for it, could not have been justified by it, and existed for a decade before anyone in the field used it seriously. If you want the mechanism at the level of the chip rather than the history, it is in why GPUs and not CPUs.