The Metrics That Matter: A Minimal LLM Dashboard

5 min read · updated August 3, 2026

Nine metrics. Each one has a name, a type, a unit and a set of labels, and each has something it deliberately does not measure. Anything a tenth metric would tell you is available from the request log when somebody asks.

Two rules before the list

Metrics are unsampled; traces are not. Everything here is a counter or a histogram, pre-aggregated, cheap to keep for a year. None of it is derived from sampled traces, because a rate computed from a 1% sample is a rate with error bars nobody drew.

Every metric is either alertable or diagnostic, and you say which. An alertable metric has a threshold and a runbook. A diagnostic metric exists to explain an alertable one. Metrics that are neither are decoration, and they are what makes dashboards unreadable.

Where the OpenTelemetry GenAI conventions already define a metric, this list uses its name — gen_ai.client.operation.duration and gen_ai.client.token.usage — so that any backend that understands the convention understands your dashboard.

1–3 · Is it working?

Health	Description
1. llm.requests (counter, {request})	Every attempt to call a model, including retries. Labels: `operation`, `provider`, `model`, `environment`, `outcome` (`ok` / `error`). Alertable as the denominator of the error rate. Excludes nothing — this is the base count, which is why the retry distinction lives in metric 3 rather than here.
2. llm.errors (counter, {error})	Failures, labelled by `error.type` using the general OpenTelemetry attribute. The taxonomy matters more than the count: separate provider 5xx, rate limits (429), context-length rejections, timeouts, and your own validation failures — they have four different owners and four different fixes. Alertable via burn rate. Excludes semantically wrong answers, which return 200 and are metric 9.
3. llm.attempts_per_call (histogram, {attempt})	How many upstream attempts one logical call took. Diagnostic. A rising mean is the earliest signal of provider trouble that failover is currently absorbing, and it explains latency and cost increases that nothing else does. Excludes application-level retries the user triggered by asking again — that is metric 9.

4–5 · The two latencies

Latency	Description
4. llm.time_to_first_token (histogram, s)	From request sent to first content byte received. Streaming only. Alertable — this is what a user experiences as responsiveness. Report p50, p95 and p99, never a mean: one cold start drags a mean past every request anyone actually had. Excludes generation time entirely, which is the entire reason it is a separate metric.
5. gen_ai.client.operation.duration (histogram, s)	Total wall clock for the operation, from the semantic conventions, with their recommended buckets growing by powers of two from 0.01s to about 82s — model latency spans four orders of magnitude and default HTTP buckets are useless for it. Diagnostic for streaming endpoints, alertable for non-streaming ones. Excludes your own pre- and post-processing; if retrieval is slow, that belongs on its own span.

Latency

Description

4. llm.time_to_first_token (histogram, s)

From request sent to first content byte received. Streaming only. Alertable — this is what a user experiences as responsiveness. Report p50, p95 and p99, never a mean: one cold start drags a mean past every request anyone actually had. Excludes generation time entirely, which is the entire reason it is a separate metric.

5. gen_ai.client.operation.duration (histogram, s)

Total wall clock for the operation, from the semantic conventions, with their recommended buckets growing by powers of two from 0.01s to about 82s — model latency spans four orders of magnitude and default HTTP buckets are useless for it. Diagnostic for streaming endpoints, alertable for non-streaming ones. Excludes your own pre- and post-processing; if retrieval is slow, that belongs on its own span.

Splitting these two is the single most valuable thing on this page for latency work. A long system prompt hurts the first and not the second; a verbose answer hurts the second and not the first; and averaging them into one “latency” number hides which problem you have.

6–7 · Money

Cost Description

6. gen_ai.client.token.usage (histogram, {token}) Tokens, split by gen_ai.token.type with values input and output. A histogram rather than a counter so you can see the distribution of prompt sizes — the p99 input length is what predicts your context-limit failures and your cost tail. Diagnostic, never alertable: token volume moves with traffic and there is no threshold that means “something is broken”.

7. llm.cost (counter, USD) Money, as a monotonic counter you take a rate of. Labels: model, provider, feature, environment — and emphatically not tenant_id, which belongs in the request log where cardinality is free. Alertable on hourly rate against an absolute ceiling; a runaway loop is the failure mode nothing else on this list detects. Excludes your observability, storage and egress costs, which are real and belong on a different dashboard.

Cost	Description
6. gen_ai.client.token.usage (histogram, {token})	Tokens, split by `gen_ai.token.type` with values `input` and `output`. A histogram rather than a counter so you can see the distribution of prompt sizes — the p99 input length is what predicts your context-limit failures and your cost tail. Diagnostic, never alertable: token volume moves with traffic and there is no threshold that means “something is broken”.
7. llm.cost (counter, USD)	Money, as a monotonic counter you take a rate of. Labels: `model`, `provider`, `feature`, `environment` — and emphatically not `tenant_id`, which belongs in the request log where cardinality is free. Alertable on hourly rate against an absolute ceiling; a runaway loop is the failure mode nothing else on this list detects. Excludes your observability, storage and egress costs, which are real and belong on a different dashboard.

On accuracy: metric 7 is only as good as your price table, so reconcile its monthly total against the provider invoice. A gap that persists means a stale rate or a code path calling a provider outside your client — both worth knowing, and invisible any other way.

8–9 · Did it do the job?

Outcome Description

8. llm.output_valid (counter, {response}) Responses that parsed and validated against the expected schema, labelled valid true/false plus feature. Alertable. For any feature with structured output this is the closest thing to a correctness signal you can compute exactly, on 100% of traffic, with no judgement in it. Excludes answers that are well-formed and wrong.

9. llm.task_outcome (counter, {task}) Your own definition of the job being done — the ticket resolved, the suggestion accepted, the document produced, or its inverse, the user regenerating immediately. Alertable only on a slow window, because it is noisy and lagging. This is the only metric on the list that requires product-specific work, and it is the one that makes the other eight worth having, because it is the denominator of cost per outcome.

Outcome	Description
8. llm.output_valid (counter, {response})	Responses that parsed and validated against the expected schema, labelled `valid` true/false plus `feature`. Alertable. For any feature with structured output this is the closest thing to a correctness signal you can compute exactly, on 100% of traffic, with no judgement in it. Excludes answers that are well-formed and wrong.
9. llm.task_outcome (counter, {task})	Your own definition of the job being done — the ticket resolved, the suggestion accepted, the document produced, or its inverse, the user regenerating immediately. Alertable only on a slow window, because it is noisy and lagging. This is the only metric on the list that requires product-specific work, and it is the one that makes the other eight worth having, because it is the denominator of cost per outcome.

If you build nothing else from this page, build metric 9. Every other number here describes the machinery; this one describes whether the machinery accomplished anything, and it is the one that lets you show spend going up and unit cost going down in the same sentence.

Labels and the cardinality budget

A metric’s cost is the product of its label cardinalities, and that multiplication is how a monitoring bill overtakes an inference bill. Compute it before you add a label rather than after.

series = operation(4) × provider(5) × model(20) × environment(3) × outcome(2)
       = 2,400 series for llm.requests.        Fine.

add feature(30)      →   72,000 series.       Acceptable, deliberate.
add tenant_id(5,000) →  360,000,000 series.   An incident, not a dashboard.
add prompt_version   →  multiply again, and it grows on every prompt edit.

Never label with an identifier. Tenant, user, request, conversation, session. These belong on spans and in the request log, where high cardinality is the whole point.
Never label with anything unbounded. Error messages (as opposed to error types), URLs with ids in them, and model names if users can supply arbitrary strings — map unknown values to other at the emit site.
Bound feature deliberately. It is worth the cardinality, which is exactly why it should be a closed enum from the attribution design rather than a free string.
Drop environment if you have separate backends per environment. A label that is constant within a backend is a multiplier for nothing.

One closing note on what is not here. There is no benchmark score, no quality index, no aggregate “model health” number. Those are composites, and a composite that moves tells you nothing about which component moved — which is precisely the property you do not want at 03:00.