How Fast Models Generate: The Roofline and What It Cannot Tell You

At batch size one, a model cannot generate faster than its memory bandwidth divided by its weight bytes. That single division predicts generation speed to within tens of per cent, explains every quantisation decision anyone makes, and is why the number a vendor quotes is usually not the number you will see.

The ceiling, in one division

Generating one token requires reading every weight the model will use for that token out of memory and into the arithmetic units. The arithmetic itself is fast; the reading is not. So the time per token is bounded below by how long it takes to stream the weights.

  tokens_per_second  <=  memory_bandwidth  /  bytes_read_per_token

  bytes_read_per_token  ~=  active_parameters  x  bytes_per_weight

For a dense model, active parameters is the whole model.

Worked: a 70B model at BF16 on an accelerator with 3.35 TB/s of
memory bandwidth (NVIDIA's published figure for the H100 SXM):

  bytes per token  =  70e9 * 2      =  1.40e11 bytes
  ceiling          =  3.35e12 / 1.40e11
                   =  23.9 tokens per second

Real systems reach roughly 60-85% of this, so expect 14-20 tokens per
second for one user on one accelerator. Which is roughly what serving
a 70B model at batch one actually does.

Notice what is absent from that calculation. The FLOP rating of the accelerator does not appear. Neither does the prompt length, the temperature, or anything about the software. At batch one it is a memory problem, and the derivation in FLOPs per answer shows the arithmetic units sit idle about 97 per cent of the time while this is happening.

Worked across models and hardware

  ceiling (tok/s)  =  bandwidth (bytes/s)  /  (params * bytes_per_weight)

Bandwidth figures below are manufacturer specifications; check the
datasheet for the exact part.

  hardware                    BW        model          ceiling
  ------------------------  --------  -------------  ----------
  H100 SXM (3.35 TB/s)      3.35e12   8B  @ BF16      209 tok/s
  H100 SXM                  3.35e12   8B  @ INT4      838 tok/s
  H100 SXM                  3.35e12   70B @ BF16       23.9 tok/s
  H100 SXM                  3.35e12   70B @ INT4       95.7 tok/s
  A100 80GB (2.04 TB/s)     2.04e12   70B @ INT4       58.3 tok/s
  consumer card (1.0 TB/s)  1.00e12   8B  @ INT4      250 tok/s
  consumer card (1.0 TB/s)  1.00e12   70B @ INT4       28.6 tok/s  *
  Apple unified (800 GB/s)  0.80e12   70B @ INT4       22.9 tok/s

  * needs 35 GB of weights; will not fit on a 24 GB card, so the
    ceiling is theoretical and the real answer is "it does not run"
    until you shard it or offload, at which point the bandwidth in
    the formula becomes the slowest link in the chain — often PCIe or
    system RAM, which is one to two orders of magnitude slower.

Two conclusions worth stating plainly. Quantisation buys generation speed in direct proportion, not as a side effect: halving bytes per weight doubles the ceiling. And the moment any part of the model spills out of accelerator memory, the relevant bandwidth becomes that of the spill path, and speed falls off a cliff rather than degrading gracefully. That is the whole content of VRAM requirements and of most disappointing local-model experiences.

Why it slows down as context grows

The weights are not the only thing read per token. The KV cache is read too, and it grows linearly with context, so a long conversation is slower per token than a short one on identical hardware.

  bytes_read_per_token  =  weights  +  kv_cache_bytes

Using the KV cache derivation from the model file size page — a
70B-shaped model with grouped-query attention at 320 KiB per token:

  at INT4 weights (35 GB) and a 2,000-token context:
    kv  =  2,000 * 327,680      =  0.66 GB
    total = 35.0 + 0.66         =  35.7 GB
    ceiling = 3.35e12 / 3.57e10 =  93.8 tok/s

  same model at a 32,000-token context:
    kv  =  32,000 * 327,680     =  10.5 GB
    total = 35.0 + 10.5         =  45.5 GB
    ceiling = 3.35e12 / 4.55e10 =  73.6 tok/s

  same model at a 128,000-token context:
    kv  =  128,000 * 327,680    =  41.9 GB
    total = 35.0 + 41.9         =  76.9 GB
    ceiling = 3.35e12 / 7.69e10 =  43.6 tok/s

Generation is 2.2x slower at 128k of context than at 2k, on the same
hardware with the same model, before any attention arithmetic is
considered.

This is the mechanical reason long conversations feel sluggish, and it is separate from the quality degradation described in context rot. Both get worse together, which makes them easy to confuse.

The batch size that changes everything

Everything above assumes one request at a time. Serving many concurrently changes the arithmetic entirely, because the weights are read once and used for every request in the batch. The crossover point — where the system stops being memory-bound and starts being compute-bound — is a property of the hardware alone.

At batch size B, per decoding step:

  FLOPs   =  2 * N * B          (each of B tokens goes through all weights)
  bytes   =  2 * N              (weights read once, shared by the batch)

  arithmetic intensity  =  FLOPs / bytes  =  B

The machine's balance point is its peak FLOP rate divided by its
bandwidth:

  H100 SXM:  989e12 FLOP/s  /  3.35e12 bytes/s  =  295 FLOP per byte

So the system is memory-bound below batch ~295 and compute-bound above.

Consequences:
  batch   1:  1 user gets ~24 tok/s;  aggregate ~24 tok/s
  batch  32:  each user still gets ~24 tok/s;  aggregate ~768 tok/s
  batch 295:  the ceiling is reached; beyond this, adding requests
              slows each of them down rather than being free.

Per-user speed is roughly constant while aggregate throughput rises
almost linearly. That is not intuition; it is the reason serving
economics work at all.

The KV cache is what actually stops you reaching batch 295 in practice: the cache is per-sequence and does not amortise, so memory capacity binds long before arithmetic does. The full picture of that trade-off is in continuous batching and throughput against latency.

Sparse models break the rule usefully

A mixture-of-experts model reads only the experts a token routes to, so the bytes-read term uses active parameters while the memory-capacity requirement uses the total.

A 400B-total, 40B-active model at INT4:

  memory needed (capacity)  =  400e9 * 0.5  =  200 GB   -> multiple cards
  bytes read per token      =   40e9 * 0.5  =   20 GB

  ceiling  =  3.35e12 / 2.0e10  =  167 tokens per second

Compare a dense 70B at INT4 on the same hardware: 95.7 tok/s.

The sparse model is nominally six times larger and generates nearly
twice as fast. It also needs six times the memory to sit in.

That is the whole commercial argument for sparse activation, expressed in the one equation where it shows up most clearly. The caveat is that expert routing varies by token, so real throughput is less predictable than the formula implies, particularly under batching where different requests in a batch want different experts.

What a quoted figure actually means

• Aggregate against per-request. A provider quoting “thousands of tokens per second” is almost always quoting aggregate throughput across a full batch. The number you experience is the per-request figure, which may be fifty times lower. Both are true; only one is about your latency.
• Tokens are not a fixed amount of text. A tokenizer with a larger vocabulary encodes the same sentence in fewer tokens, so 100 tokens per second from two providers can be materially different amounts of English per second. When comparing across providers, convert to characters or words per second using the ratio from the token counts page.
• Time to first token is a separate number. It is dominated by prefill, which is compute-bound rather than bandwidth-bound, so it obeys none of the arithmetic above. See time to first token against tokens per second.
• Means hide the distribution. Generation speed under production load varies with what else is in the batch. A p50 and a p95 describe the experience; a mean describes nothing, as latency percentiles sets out.
• Reasoning models produce many tokens you do not see. Time to a useful answer is the metric, and tokens per second can be excellent while it is poor.
• The one benchmark with full disclosure. MLCommons publishes MLPerf Inference results with the complete system configuration, the scenario and the latency constraints attached. When you need a comparable third-party throughput number rather than a marketing one, that is the shape of thing to look for.