Skip to content
Multigrid

AI hardware, GPUs & compute

The arithmetic that decides what a model costs to run: memory bandwidth, VRAM, interconnect and utilisation, derived rather than benchmarked.

Hardware pages go stale faster than anything else written about AI. A card’s specification is superseded, a rental rate moves, an export rule changes, and the page is quietly wrong while still reading as confident. So these pages are built out of the relationships that do not move: memory bandwidth divided by bytes read per token bounds decode speed; parameters times bytes per parameter plus the KV cache bounds what fits; interconnect bandwidth and latency bound how far a model can be split.

Every formula here is stated with its terms defined and its assumptions written down, so you can put today’s datasheet and today’s quoted price into it and get an answer that is current even though the page is not. Where a specification is named it is attributed and dated and used as an example input. Nothing here was benchmarked by us.

Why GPUs and Not CPUs

The two properties that make a GPU the right shape for a transformer — wide parallelism and high memory bandwidth — with the arithmetic that shows which one actually binds.

5 min read

VRAM Requirements: How Much Memory for Which Model

The complete memory formula — weights, KV cache, activations and runtime overhead — with each term derived and worked through at four quantisation levels.

5 min read

Memory Bandwidth Is the Real Bottleneck

Why generation speed tracks bandwidth rather than FLOPs, derived from arithmetic intensity, including the batch size at which the bottleneck flips.

5 min read

Comparing Datacentre GPUs for Inference Without Trusting the Spec Sheet

A procedure for ranking accelerators on the numbers that decide inference cost, using this quarter's datasheets and the rate you were actually quoted.

5 min read

Consumer GPUs for Local LLMs: What Fits in Each VRAM Tier

What each common VRAM tier can hold, derived from the memory formula rather than reported, plus the speed that tier's bandwidth implies.

5 min read

Apple Silicon for Local Inference: Capacity Against Bandwidth

How unified memory changes the two terms that matter — it buys capacity a discrete GPU cannot, and gives back bandwidth — worked as arithmetic on both sides.

5 min read

TPUs, Trainium and Custom Accelerators: Where Non-GPU Silicon Wins

The structural reasons a fixed-function accelerator can beat a GPU, the compiler assumptions that make it possible, and what you give up in exchange.

5 min read

GPU Cloud Pricing: Building the Cost Model Instead of Reading the Sticker

The formula for effective hourly cost — reservation, storage, egress, idle and the utilisation divisor — with every term named and none of them invented.

5 min read

GPU Utilisation: Why the Number Is Lower Than You Think

What the utilisation percentage actually measures, why it is not efficiency, and how to find the bubbles in a serving stack.

4 min read

Multi-GPU Inference: Tensor and Pipeline Parallelism

What each parallelism strategy splits, what it costs in communication, and the arithmetic for when splitting a model helps latency and when it only helps capacity.

5 min read

The Interconnect: Why Communication Is the Hidden Scaling Limit

How the two numbers that describe a link — bandwidth and latency — turn into a floor on per-token time, and why the hierarchy inside a rack exists.

5 min read

CUDA and the Software Moat: What It Is Actually Made Of

The five layers a competing platform has to replace, why the top ones are harder than the bottom ones, and a checklist for evaluating portability on the day you read this.

4 min read

Flash Attention: IO-Awareness Explained Without the Kernel Code

Why the standard attention implementation is limited by memory traffic rather than arithmetic, and how tiling with an online softmax removes the quadratic memory term.

5 min read

Serving Engines Compared: The Features That Decide Throughput

The five mechanisms that separate a fast inference server from a slow one, and a benchmarking protocol with the confounders named.

4 min read

Training Compute vs Inference Compute: Where the Crossover Is

The two standard FLOP formulas, and the derivation showing that inference overtakes training once a model has served roughly three times its training tokens.

5 min read

What a Frontier Training Run Costs: The Methodology, Not the Headline

How to estimate a training run's compute cost from parameters, tokens and utilisation — every step shown, with the price left as your input.

5 min read

Edge Inference: What Actually Fits on a Phone

The memory budget, the bandwidth ceiling and a lower bound on energy per token, derived for mobile-class hardware.

5 min read

The GPU Shortage: Why Accelerator Supply Is Structurally Inelastic

The mechanisms that make accelerator supply slow to respond to demand, and the primary sources to check for the situation on the day you read this.

4 min read

Renting vs Buying GPUs: The Break-Even, Derived

The total-cost-of-ownership formula with every term named, solved for the utilisation at which owning beats renting — using your quotes, not ours.

6 min read

Cooling, Power Density and the Physical Limits of a Rack

Why the rack, not the chip, is the binding constraint on AI capacity — derived from the heat-transfer arithmetic that makes air cooling run out.

5 min read

Other topics

AI hardware, GPUs & compute · Multigrid