What GPU Utilization Percent Actually Means During Local Inference

nvidia-smi says 100% and the model produces twenty tokens a second. Both readings are correct, and reconciling them requires knowing what the utilisation counter is counting — which is time, not work.

What the number is defined as

NVIDIA’s management library defines GPU utilisation, exposed as utilization.gpu, as the percent of time over the past sample period during which one or more kernels was executing on the GPU, with the sample period between one second and one sixth of a second depending on the product. Read that definition literally, because every misunderstanding of the metric comes from reading it loosely.

• It counts time, not work. A kernel that does nothing but wait on memory counts exactly as much as one saturating the tensor cores.
• It says one or more. A single kernel occupying a single streaming multiprocessor out of the hundred-odd on the die registers the same as one filling every SM.
• It is a duty cycle over a sample window. A workload issuing a kernel every millisecond reads as continuously busy even if each kernel is trivial.

The neighbouring metric catches people just as often. utilization.memory is also a time metric — the percent of time during which device memory was being read or written — and is not how much VRAM is in use. The occupancy figure people actually want there is memory.used against memory.total, which is a different query:

nvidia-smi --query-gpu=utilization.gpu,utilization.memory,memory.used,memory.total \
  --format=csv -l 1

Busy is not the same as fast

Token generation at batch one is a sequence of matrix-vector products. For each weight matrix, every weight is read out of memory, multiplied by one activation, and added into an accumulator. Each weight is used exactly once and then discarded — there is no reuse to exploit, because there is only one input vector to multiply it by.

The hardware consequence is that the SMs spend nearly all their time waiting for data to arrive. They are, in the counter’s sense, executing a kernel the whole time, so utilisation reads high. They are also nearly idle in the sense you care about, because the arithmetic units have very little to do between loads. The number is not lying; it is answering a different question from the one you asked.

The clearest demonstration that the metric is insensitive to how much work is happening is that the two phases of a single request produce nearly the same reading. Prefill processes every prompt token in parallel, so each weight it loads is reused across the whole micro-batch and the tensor cores are genuinely the constraint. Decode then produces one token at a time with no reuse at all. Those two regimes differ by roughly two orders of magnitude in arithmetic performed per byte moved, and utilization.gpu reports both as busy. Watch it at one-second intervals during a request with a long prompt and a short answer, and then one with a short prompt and a long answer: the line is flat in both, while the tokens-per-second you experience is not.

The arithmetic behind 100% and slow

The size of the gap can be derived from published specifications. The relevant quantity is arithmetic intensity: floating-point operations performed per byte moved from memory.

Decode, batch 1, FP16 weights:
  work per weight  = 1 multiply + 1 add  = 2 FLOPs
  bytes per weight = 2
  arithmetic intensity = 2 / 2 = 1 FLOP per byte

The hardware's balance point (its "ridge"), from NVIDIA's
published H100 SXM figures — 989.5 dense FP16 TFLOP/s
(1,979 with sparsity, halved) and 3.35 TB/s of HBM3:

  ridge = 989.5e12 FLOP/s / 3.35e12 B/s
        = 295 FLOPs per byte

  achievable share of peak compute at 1 FLOP/byte
        = 1 / 295
        = 0.34%

Under those assumptions, single-stream decoding can use about a third of one percent of the card’s peak arithmetic while the utilisation counter reads 100%. Quantising to four bits raises intensity to roughly 3.3 FLOPs per byte, which is better by a factor of three and still two orders of magnitude below the ridge. The conclusion is not that something is misconfigured. It is that decoding is a memory-bandwidth workload running on a machine designed for arithmetic, and no setting changes that.

The number that actually bounds you

Since every weight is read once per token, the tokens-per-second ceiling is a division:

tokens/s  <=  memory_bandwidth / bytes_of_weights_read_per_token

An 8B model at ~4.8 bits per weight is ~4.8 GB of weights:

  3.35 TB/s  ->  3350 / 4.8  =  ~698 tokens/s
    900 GB/s ->   900 / 4.8  =  ~188 tokens/s
    360 GB/s ->   360 / 4.8  =   ~75 tokens/s
     50 GB/s ->    50 / 4.8  =   ~10 tokens/s   (typical of system RAM)

Those are upper bounds from that division, not predictions: they ignore KV-cache reads, which grow with context, and all kernel launch and sampling overhead, so real rates land below them. Their value is diagnostic. If you are getting a third of the ceiling, there is something to fix. If you are getting eighty percent of it, the machine is doing what the physics allows and the only remaining levers are fewer bytes per token or a different machine. Measuring your actual rate is the other half of that comparison, and the last line of the block is why CPU-only inference behaves the way it does — system RAM bandwidth is roughly an order of magnitude below a discrete card’s.

Metrics that answer the question you had

NVIDIA’s Data Center GPU Manager exposes profiling fields that are defined as ratios of cycles rather than as duty cycles over kernels, and they separate the things utilization.gpu conflates.

• DCGM_FI_PROF_SM_ACTIVE — the ratio of cycles for which an SM has at least one warp resident, averaged over all SMs. Unlike the nvidia-smi figure, this falls when most of the die is idle.
• DCGM_FI_PROF_DRAM_ACTIVE — the ratio of cycles for which the device memory interface is sending or receiving data. This is the metric that confirms the diagnosis: near 1.0 during decode means the card is moving data as fast as it can and there is no headroom to recover.
• DCGM_FI_PROF_PIPE_TENSOR_ACTIVE — how busy the tensor pipes are. Low during decode and high during prefill, which is the clearest single demonstration that the two phases are different workloads on the same hardware.
• Tokens per second. The only metric that is the thing you care about rather than a proxy for it. Everything above exists to explain a tokens-per-second number you have already measured.

Practical readings. Utilisation at 100% with slow generation is normal, and the remedies are all “move fewer bytes per token”: a smaller model, a lower bit width, speculative decoding, or serving several requests concurrently so each weight read serves more than one token. Utilisation well below 100% during generation is the more interesting reading, because it means the GPU is idle some of the time and the bottleneck is elsewhere — tokenisation, the client, a slow HTTP path, or layers running on the CPU because the model did not fit. That case, and why it feels different from a hosted endpoint, is covered in why a local model feels slower than the cloud.