Comparing Datacentre GPUs for Inference Without Trusting the Spec Sheet

Any page that ranks accelerators by name is wrong within a product cycle, and most of them were wrong on publication because they ranked by peak FLOPs. What survives is the procedure. Here it is, with the arithmetic worked through symbolically so you can put today’s numbers in.

Why there is no ranking table here

Two reasons, and they are different. The first is staleness: parts, availability and quoted rates all move faster than an article gets revised, so a table is a liability rather than an asset. The second is that a ranking hides the thing that decides the answer — the ranking depends on your model, your context length and your utilisation, and any two of those can reverse it.

So this page does not tell you which accelerator is best. It tells you which four numbers to take off the vendor’s datasheet, in what order, and how to turn them into a cost per million tokens you can compare against any alternative including a metered API.

Before any of it, decide which question you are asking, because there are two and they have different winners. Latency asks how fast one sequence can be generated, and it is answered almost entirely by memory bandwidth. Throughput asks how many tokens per second the device can produce across all users, and it is answered mostly by how large a batch you can hold, which is a memory capacity question. Most published comparisons quietly answer one and are read as answering the other.

The four numbers that decide it

Note what is missing: clock speed, core count, and the marketing composite figures. Those are inputs to number three, and number three is third for a reason.

Two more numbers belong on the sheet without being part of the ranking. Rated power decides whether a device can be deployed in the facility you have and, if you are renting, is already inside the rate. And the numeric formats with hardware support decide which value of b you may actually use — a device without fast 8-bit or 4-bit paths forces you to a larger b, which halves your decode ceiling and doubles your memory footprint compared with one that has them. That is a larger effect than most of the differences people rank on, and it is visible on the datasheet before you rent anything.

The procedure

• Step 1 — fit. Compute the memory budget for your model at your intended context and batch. Any device whose capacity is below that is out, or moves into the multi-device comparison, where you add the interconnect term and a parallelism efficiency factor below one.
• Step 2 — decode ceiling. tok/s ≤ BW / (P × b), per device. Multiply by the batch you can actually hold in the memory left over after weights, because aggregate throughput is what you are buying.
• Step 3 — apply a realisation factor. Kernels do not hit peak bandwidth. Applying a factor in the region of 0.6–0.8 to the ceiling is a defensible planning assumption; state it as an assumption rather than pretending it is a measurement, and replace it with your own figure as soon as you have run the workload once.
• Step 4 — divide by the rate you were quoted. Not a list price you found; the rate for the commitment you are actually able to make.

cost_per_1M_output_tokens
    = rate_per_hour / (aggregate_tokens_per_second * 3600 / 1e6)

aggregate_tokens_per_second
    = realisation * BW / (P * b) * B_effective

A worked comparison

Two hypothetical devices, and the point is the shape of the answer, not the devices. Device A: 80 GB, 3 TB/s. Device B: 192 GB, 4.5 TB/s at 1.6× device A’s hourly rate. Model: 70B at fp8, so weights are 70 GB.

• Fit. Device A holds the weights with 10 GB left — enough for only a handful of concurrent 8k sequences at roughly 0.16 MiB per token with an fp8 cache. Device B leaves 122 GB, room for a large batch.
• Ceiling. A: 3000/70 ≈ 43 tok/s per sequence. B: 4500/70 ≈ 64 tok/s. Only a 1.5× difference per stream.
• Aggregate. But B holds perhaps ten times the batch, and below the ridge point batching is nearly free. The aggregate gap is far larger than the per-stream gap, which at a 1.6× rate makes B cheaper per token despite costing more per hour.

The general lesson generalises past the invented numbers: for throughput serving, memory capacity often beats bandwidth, because capacity buys batch and batch is the free multiplier below the ridge. For latency-sensitive single-stream work the ranking flips to pure bandwidth. Two different jobs, two different winners, same two devices.

Traps in the datasheet

• Sparsity-assisted FLOPs. Some peak figures assume a structured sparsity pattern the model must actually have. If yours does not, halve the number.
• Precision shell games. A headline throughput at 4-bit next to a bandwidth figure is comparing two different configurations. Fix the precision first, then read both numbers.
• Aggregate versus per-device. Bandwidth and capacity quoted for a whole eight-device system tell you nothing about whether one model fits one device.
• Power and cooling as a hidden constraint. A device you cannot supply or cool at its rated power will clock down, and the datasheet figure quietly stops applying. If you are racking hardware rather than renting it, that constraint is upstream of all four numbers.
• Bandwidth quoted as peak, never as achieved. Real kernels realise a fraction of the theoretical figure and the fraction varies with access pattern. Two devices with identical headline bandwidth can differ in what a decode kernel actually gets out of them, which is one of the few places where measurement genuinely beats derivation — and where you should trust your own measurement over anyone else’s.

A final note on how to use the result. This procedure gives you a ranking that is correct for one model at one context length and one concurrency, which is exactly as general as your deployment is. That is a feature: it makes the answer honest and it makes it recomputable. When the model changes or the traffic shape changes, run it again — the inputs are four datasheet numbers and one quote, and the whole calculation takes minutes rather than the days a benchmark would.