TPUs, Trainium and Custom Accelerators: Where Non-GPU Silicon Wins

A GPU is a general parallel computer that happens to be excellent at matrix multiplication. A purpose-built accelerator is a matrix multiplier that happens to be programmable. Everything that follows — the efficiency, the compiler, the sharp edges — comes out of that one difference.

Why non-GPU silicon exists at all

Generality costs area and power. A GPU carries instruction fetch and decode for every warp, a register file sized for arbitrary programs, caches with coherence machinery, schedulers for divergent control flow, and the whole graphics heritage. None of that is needed to multiply two matrices. If you know in advance that the workload is a fixed sequence of large dense multiplications with a handful of element-wise operations between them, you can spend the reclaimed area on more multiply-accumulate units and more on-chip memory.

The prize is performance per watt and per unit of silicon, in a domain where power delivery and cooling are hard limits rather than preferences. Whoever runs enough of one workload to justify the design cost has an argument for doing it, which is why the accelerators that exist are largely built by the organisations with the largest internal demand.

The economics only close at scale, and it is worth being explicit about why. Designing and taping out a chip on a leading process is a large fixed cost, and it must be recovered across the units built. An organisation deploying a very large fleet of one workload amortises it easily; a vendor selling to a fragmented market cannot, which is the same reason general-purpose parts dominate everywhere else in computing. The corollary matters for a buyer: custom silicon tends to exist as capacity inside a platform rather than as a part on a price list, so evaluating it is usually evaluating a service.

Systolic arrays and dataflow

The recurring architectural idea is the systolic array: a grid of multiply-accumulate cells where each cell passes its operand to its neighbour rather than writing back to memory. Data flows through the grid in a rhythm, hence the name, and each value loaded from memory is reused across the whole row or column it passes through.

Read that against arithmetic intensity and the point becomes exact: a systolic array raises the FLOPs performed per byte moved, which is precisely the ratio that determines whether the memory system or the arithmetic units bind. A design that reuses operands in the fabric can sit further up the roofline on the same memory bandwidth. Large on-chip memory serves the same end from the other direction — bytes that never leave the chip do not consume external bandwidth, and they cost markedly less energy to access than bytes from DRAM.

The catch is symmetric with the benefit. A systolic array is efficient when the matrices are large and their shapes are known so the mapping can be planned. Small matrices, ragged shapes and data-dependent control flow leave cells idle, and idle cells in a fixed grid cannot be repurposed the way GPU lanes can.

Decode is precisely the awkward case. Generating one token multiplies a vector by each weight matrix, and a vector is a matrix with one row, so a large grid designed to consume tiles has very little to consume. Batching restores the rows and with them the efficiency, which means a fixed-function accelerator’s advantage is largest exactly where throughput serving lives and smallest at batch size one. That is a consistent structural result rather than a property of any generation, and it is the single most useful thing to know when reading a claim about one.

The compiler is the product

On a GPU you can write a kernel. On most custom accelerators you cannot meaningfully hand-write the inner loop; you express the computation as a graph and a compiler maps it onto the fabric — laying out the tiles, scheduling the data movement, fusing the element-wise operations, deciding what stays on-chip.

This is why the compiler, not the chip, is what you are really evaluating. Ask the questions the architecture makes load-bearing:

• Does it need static shapes? Ahead-of-time compilation for a fixed shape is where much of the efficiency comes from. Serving has inherently dynamic shapes — variable prompt lengths, variable batch — so the runtime must bucket them, and bucketing wastes work at the edges of each bucket.
• What happens to an unsupported operation? A graceful answer is a fallback path. An ungraceful one is a compile error, and you are rewriting a model to suit a compiler.
• How long does compilation take, and when does it happen? A recompilation triggered by a new input shape at request time is a latency spike with a cause that is invisible in the application.
• Are the attention kernels first-class? The IO-efficient attention implementations that make long context affordable are hand-tuned per architecture. Whether an equivalent exists for a given accelerator is a concrete, checkable question.

What you give up

Deciding without a benchmark you cannot run

The honest procedure is narrow, and it is narrow because the useful comparisons require access most readers do not have.

• Check that your op set compiles before anything else. This is a yes/no you can establish in a day, and it decides more outcomes than any throughput number.
• Compare on delivered cost per token, not on peak figures. Custom accelerators are usually sold as a managed service rather than a part, so the number available to you is a rate per hour or per token — which is the number that belongs in the comparison anyway.
• Weight availability heavily. Silicon you can obtain at the capacity you need beats silicon you cannot, and this is the variable that moves fastest and that no article can tell you.
• Assume the shape of the trade is stable and the numbers are not. Fixed-function silicon wins on efficiency for steady-state, large, well-supported workloads, and loses on flexibility. That has been true across generations; every specific figure attached to it has not.