When a GPU Supports Both INT8 and FP8, Which One Wins

Both formats put a weight in eight bits and both halve the memory traffic against FP16. The question of which wins has three different answers depending on whether you mean accuracy, peak arithmetic rate, or what your runtime will actually execute — and on the hardware where both exist, the second of those is a tie.

What the two formats actually are

INT8 is an eight-bit signed integer plus a scale. The real value is recovered as value = scale x q, with the scale held per tensor, per channel or per small block. All 256 codes are evenly spaced, so the resolution inside the represented range is uniform and the range itself is whatever the scale says. Choosing that scale is the entire problem: it has to cover the largest magnitude present, and everything smaller shares the resolution that is left.

FP8 is a floating-point format and comes in two encodings. E4M3 has four exponent bits and three mantissa bits; E5M2 has five and two. Both spend bits on an exponent, so the spacing between representable values grows with magnitude — fine resolution near zero, coarse resolution far from it, and a dynamic range far wider than eight uniform steps could span.

That difference is the whole accuracy story, because transformer activations are not evenly distributed. A small number of channels carry values one or two orders of magnitude larger than the rest, and those outliers set the scale for a per-tensor INT8 quantiser. Once they do, the ordinary channels are left with a fraction of the available codes. The mitigations are well known — per-channel or per-group scales, or keeping the outlier channels in higher precision — but they are mitigations, and FP8’s exponent absorbs the same spread without any of them. This is why FP8 is usually the easier format to quantise activations into, and why weight-only INT8 remains perfectly respectable: weights are much better behaved than activations.

Which tensor cores accelerate which

Native FP8 tensor-core support begins at CUDA compute capability 8.9. Ada Lovelace (8.9) and Hopper (9.0) have it; Ampere (8.0 and 8.6) does not, despite having perfectly good INT8 tensor cores. vLLM documents this boundary directly in its FP8 quantisation guide: full W8A8 FP8 — eight-bit weights and eight-bit activations — requires compute capability 8.9 or newer, while Ampere is supported for weight-only schemes where the multiply happens at higher precision.

The consequence for a machine with an Ampere card is specific and worth stating plainly: an FP8 checkpoint will usually still run, because the runtime converts the weights up to FP16 before the matrix multiply. You keep the memory saving on the weights, which is the part that matters most for single-stream decoding, and you get none of the arithmetic speedup, because the arithmetic is happening in FP16. An INT8 path on the same card does have hardware behind it.

At the other end, Blackwell adds four-bit tensor-core formats, and NVIDIA has described the Blackwell Ultra tensor cores as trading rate in other formats — INT8 and FP64 among them — for a large gain in NVFP4. The lesson is that “newer silicon is faster at everything” stopped being true: each generation optimises the format it expects you to use, and an older format can get slower in absolute terms.

The peak numbers, and why they are equal

NVIDIA publishes peak tensor-core rates per format. For the H100 SXM, its specification table gives FP16 Tensor Core at 1,979 teraFLOPS, FP8 Tensor Core at 3,958 teraFLOPS and INT8 Tensor Core at 3,958 TOPS, with the footnote that these figures are stated with 2:4 structured sparsity. Halving them for the dense case that ordinary inference runs in:

H100 SXM, dense (published figures / 2):

  FP16 Tensor Core   989.5 TFLOP/s
  FP8  Tensor Core  1979   TFLOP/s   = 2.0x FP16
  INT8 Tensor Core  1979   TOP/s     = 2.0x FP16

  FP8 : INT8 ratio  =  1979 / 1979  =  1.00

A100 80GB SXM, dense:

  FP16 Tensor Core   312 TFLOP/s
  INT8 Tensor Core   624 TOP/s       = 2.0x FP16
  FP8                — no row exists —

The derived answer to “which is faster” is therefore: on hardware that accelerates both, neither. They occupy the same tensor-core datapath at the same rate, both at twice the FP16 rate. On hardware that accelerates only one, the one it accelerates is faster by a factor of two, and which one that is depends entirely on the generation. Sources: NVIDIA’s H100 product specifications and the A100 datasheet.

Why the peak gap barely matters at batch one

All of the above is about arithmetic throughput, and a local model answering one request at a time is not limited by arithmetic throughput. Generating a token is a sequence of matrix-vector products: each weight is read from memory, used in one multiply-add — two floating-point operations — and not reused. So the work per byte read is fixed and tiny, and the runtime is set by how fast weights can be pulled out of memory.

Under that regime, what an 8-bit format buys you is bytes, and both formats buy the same bytes. Halving the weight size against FP16 roughly doubles the ceiling on tokens per second, whether the eight bits are integer or floating-point. The 2× tensor-core advantage shows up in prefill, where a long prompt is processed in parallel and the arithmetic units are genuinely the constraint, and in serving many concurrent requests, where batching restores reuse. It does not show up in a single-user chat. What GPU utilisation percent means during inference works that regime out in detail.

How the decision actually gets made

• By what your runtime supports. This dominates everything else. llama.cpp’s eight-bit type, Q8_0, is a block-scaled integer scheme — it is not the vendor FP8 format, and a GGUF build has no FP8 tensor path to reach for. If you are running GGUF, the INT8-family question is the only one on the table.
• By what the weights were published as. Re-quantising an already-quantised checkpoint into a different eight-bit format compounds error rather than replacing it — see requantizing an already-quantized model. Prefer a checkpoint published in the format your hardware accelerates over converting one that was not.
• By the accuracy you can verify. The outlier argument above says FP8 activations are easier to get right than INT8 activations. It does not say your model, at your task, loses measurably either way. That is a thing to check on your own evaluation set rather than infer from a format.
• Not by eight bits at all, quite often. On a memory- constrained local machine the interesting comparison is usually between eight bits and four, not between two eight-bit encodings — quantisation level by use case is where that trade is made.