Monitoring GPUs in Production

Most GPU dashboards show utilisation, temperature and memory used, and none of the three tells you whether your inference service is healthy. The counters that do — throttle reasons, ECC and XID events, power draw against the enforced limit — are less prominent and more informative, because each of them names a physical condition that precedes a user-visible failure.

Where the numbers come from

There is one underlying source and several ways to read it. NVIDIA exposes device telemetry through a management library; the nvidia-smi command is a thin front end to it, and the Data Center GPU Manager (DCGM) is the daemon designed to be scraped, with an exporter that publishes the same fields in Prometheus format.

# One-off inspection, human readable
nvidia-smi

# The version to use in a script: pick fields, no header, machine parseable
nvidia-smi --query-gpu=index,name,utilization.gpu,utilization.memory,\
memory.used,memory.total,temperature.gpu,power.draw,power.limit,\
clocks_throttle_reasons.active,ecc.errors.uncorrected.volatile.total \
  --format=csv,noheader,nounits

# Per-process attribution: which PID is holding device memory
nvidia-smi --query-compute-apps=pid,process_name,used_memory \
  --format=csv

# Continuous sampling during an incident, one line per second
nvidia-smi dmon -s pucvmet -d 1

For a Kubernetes cluster, run the DCGM exporter as a DaemonSet on GPU nodes and scrape it; it labels metrics with the pod and container using each device, which is what makes per-workload attribution possible. Field names in that exporter follow the pattern DCGM_FI_DEV_* — for example DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_POWER_USAGE. Check the exporter’s current field list for exactly which are enabled in your build, because the default set is a subset of what the library can report.

Utilisation is not a workload metric

The GPU utilisation figure is defined as the percentage of the last sampling period during which at least one kernel was executing on the device. Read that definition carefully: it counts time with any work, not fraction of the device doing work.

A kernel using two per cent of the streaming multiprocessors, looping continuously, reports 100%. A perfectly saturated device also reports 100%. The metric cannot distinguish them, which means it cannot tell you whether you have headroom, and it cannot tell you whether you are in trouble — as GPU autoscaling argues at more length, a saturating signal cannot drive a controller.

What it is good for: detecting a device doing nothing. A serving pod whose GPU utilisation is near zero while requests are queueing is a real and useful alert, because it means work is stuck somewhere else — the tokeniser, a lock, a data loader, a network call. What GPU utilisation actually measures goes deeper into the counter.

The counters that mean something

The general principle behind that list: prefer counters that name a condition over counters that report a level. A throttle reason, an ECC event and an XID are conditions — they mean something specific has happened. Temperature and utilisation are levels whose interpretation depends entirely on context.

Service-level metrics beat device metrics

Device counters explain problems. Service metrics detect them. If you have limited alerting attention, spend it here first:

• Queue depth and queue wait per replica. The earliest indicator of capacity trouble, and the one used for autoscaling.
• Tokens per second per replica. The productivity number. A gradual decline at constant load is the signature of thermal throttling, a degraded card, or a change in the input distribution towards longer prompts.
• Time to first token, p95. What users feel, and it rises before end-to-end latency does.
• Batch size actually achieved. If your server batches, this reveals whether the batching is working. A collapse in batch size at constant request rate means requests are not overlapping in the way the throughput calculation assumed.
• Failed generations by cause. Out of memory, timeout, device error, truncation. Each points somewhere different.

Emit all of these with a device identifier and a node label so that a service-level anomaly can be joined to a device-level cause in one query. That join is the entire value of collecting both, and it is worth designing the labels for it deliberately — the minimum set of LLM metrics covers the application side.

Three alerts worth having

Alert on symptoms a human must act on, and keep the list short enough that every page is believed. Three pass that test.

groups:
- name: gpu-serving
  rules:

  # 1. The users' symptom. Everything else is diagnosis.
  - alert: InferenceQueueWaitHigh
    expr: histogram_quantile(0.95,
            sum by (le, service) (rate(inference_queue_wait_seconds_bucket[5m]))
          ) > 10
    for: 10m
    labels:   { severity: page }
    annotations:
      summary: "p95 queue wait above 10 s for 10 minutes on {{ $labels.service }}"
      runbook: "https://runbooks.example.com/ai/latency"

  # 2. Hardware saying something is wrong. Rare, and always real.
  - alert: GPUDeviceFault
    expr: increase(DCGM_FI_DEV_XID_ERRORS[10m]) > 0
       or increase(DCGM_FI_DEV_ECC_UNCORRECTABLE_TOTAL[10m]) > 0
    for: 0m
    labels:   { severity: page }
    annotations:
      summary: "XID or uncorrected ECC on {{ $labels.instance }} gpu {{ $labels.gpu }}"
      runbook: "https://runbooks.example.com/ai/gpu-fault"

  # 3. Capacity is silently disappearing: paying for compute not being delivered.
  - alert: GPUThrottledSustained
    expr: avg_over_time(DCGM_FI_DEV_CLOCK_THROTTLE_REASONS[30m]) > 0
    for: 30m
    labels:   { severity: ticket }
    annotations:
      summary: "GPU throttled for 30 min on {{ $labels.instance }} — check cooling and power cap"

Everything else — temperature, utilisation, memory used, power — goes on a dashboard, not into a pager. And give the first alert a multi-window burn rate against a stated objective rather than a bare threshold if you have an error budget defined; SLOs for AI services covers how to set the objective, and alerting on LLM systems covers what belongs in a page versus a ticket.

The dashboard, in one screen

One screen, read top to bottom, arranged so that the eye travels from symptom to cause.

1. Row one, service: request rate, TTFT p50 and p95, queue wait p95, error rate by class. This row answers “is anything wrong?”
2. Row two, throughput: tokens per second per replica, achieved batch size, concurrent generations against the configured maximum. This row answers “is the work getting through?”
3. Row three, devices: per-GPU framebuffer free, throttle reason as a state timeline rather than a line chart, power draw against limit. This row answers “why not?”
4. Row four, events: XID and ECC counts, pod restarts, deploy markers. This row answers “what changed?” and a deploy marker on the same time axis resolves a surprising fraction of incidents on its own.

Add one number that is neither a device nor a service metric: cost per thousand successful requests, over time. It is the one line on the dashboard that a non-engineer will read, and it catches regressions the others do not — a change that improves latency by adding replicas shows up here and nowhere else. Cost allocation across teams covers how to compute it per team.