Alerting on LLM Metrics Without Alarm Fatigue

Most LLM alerting starts as a threshold on latency and a threshold on error rate, fires nine times in the first week, and is muted by the second. The fix is not better thresholds. It is a different trigger model and a much shorter list of things allowed to page.

Level-triggered, not edge-triggered

An edge-triggered alert fires on a transition: latency crossed 3 seconds, error rate spiked. It is easy to write and it is why your phone buzzed at 03:00 about a condition that resolved itself in forty seconds. A level-triggered alert asks a different question — is the system currently in a bad state, and has it been for long enough to matter?

Concretely, the difference is that the alert condition is evaluated over a window and describes a sustained state, and it clears when the state clears rather than when someone acknowledges it. Every rule below is of that shape. Anything that fires on a single scrape does not belong in a paging policy; put it in a dashboard.

Page on symptoms, ticket on causes

The reliable partition, straight out of ordinary SRE practice and entirely applicable here:

• Page when users are being harmed now, and a human can do something about it in minutes. That is a small list: the feature is failing, the feature is unusably slow, or money is leaving the building at an unplanned rate.
• Ticket when something is degraded, trending wrong, or will bite in days. Rising retry rate. One provider slower than usual while failover is absorbing it. Attribution coverage slipping.
• Neither for everything else. If nobody would act on it, it is a chart.

The distinction matters more for LLM features than for a normal service because so many of the interesting signals are causes: a provider 429 rate, a fallback rate, a cache-hit drop. If failover is working, none of those are user-visible and none of them should wake anyone. They are exactly what you want in the morning ticket queue.

Burn-rate alerts, with the numbers

The standard design — described in Google’s Site Reliability Workbook chapter on alerting on SLOs — is to alert on how fast you are consuming the error budget rather than on the raw rate. Burn rate 1 means you will exactly exhaust the budget at the end of the period; burn rate 14.4 over an hour means you have burned 2% of a 30-day budget in that hour.

Two windows per rule, a long one for signal and a short one so the alert resolves promptly. The canonical ladder, for a 30-day SLO window:

Requiring both windows to breach is what removes the false positives: a two-minute blip does not move a one-hour window, and a resolved incident clears the five-minute window immediately instead of leaving the page hot for an hour.

The property that makes this design worth the setup is that all four rules take a single parameter — the error budget from your SLO — so tightening or loosening the objective adjusts every threshold coherently. Compare that to a set of hand-tuned thresholds, where changing the target means revisiting each rule and hoping they remain consistent with one another.

It also scales sensibly with traffic in a way absolute thresholds do not. A 5% error rate on a feature doing ten requests a minute is usually noise; on one doing ten thousand it is an outage. Burn rate expresses both as the same quantity, so the same rule works for a new feature and a mature one without anybody remembering to retune it as volume grows.

# PromQL, for an SLO of 99.5% successful LLM requests.
# error_budget = 1 - 0.995 = 0.005

- alert: LLMFastBurn
  expr: |
    (
      sum(rate(llm_requests_total{status="error"}[1h]))
        / sum(rate(llm_requests_total[1h])) > (14.4 * 0.005)
    )
    and
    (
      sum(rate(llm_requests_total{status="error"}[5m]))
        / sum(rate(llm_requests_total[5m])) > (14.4 * 0.005)
    )
  for: 2m
  labels: { severity: page }

- alert: LLMSlowBurn
  expr: |
    (
      sum(rate(llm_requests_total{status="error"}[3d]))
        / sum(rate(llm_requests_total[3d])) > (1 * 0.005)
    )
    and
    (
      sum(rate(llm_requests_total{status="error"}[6h]))
        / sum(rate(llm_requests_total[6h])) > (1 * 0.005)
    )
  for: 1h
  labels: { severity: ticket }

The four LLM-specific pages

Beyond availability and latency, which you already alert on for everything else, four conditions are specific to model-backed features and genuinely warrant waking someone.

• Spend rate. A runaway agent loop or a prompt that suddenly retrieves ten times as much context can multiply your bill within hours, and nothing else in your monitoring will notice. Alert on rolling hourly spend against a ceiling you set — an absolute number, not a percentage, because percentages of a small baseline fire constantly. Pair it with a hard spend cap that acts without a human.
• Structured-output validity. If a downstream system parses the model’s output, schema-validation failure rate is a user-facing error rate that returns HTTP 200. It belongs in the SLI and therefore in the burn-rate rule above.
• Total provider failure with no fallback left. Failover exhaustion is different from failover happening. The first is a page; the second is a ticket. Distinguish them explicitly, or you will page on every routine provider wobble.
• Guardrail block rate. A safety filter or validator that starts rejecting a large share of legitimate traffic is a complete outage that looks, from the error rate, like a healthy service refusing to answer. Alert on both directions of this rate.

What must never page

One more discipline that costs nothing: every alert needs a runbook link in its annotation, and the runbook needs to say what the alert means and what to check first. An alert nobody knows how to action gets muted, and a muted alert is worse than no alert because it creates the belief that something is watching.

The other habit worth adopting is to review fired alerts on a cadence — monthly is enough — and ask one question of each: did anyone do anything about it? An alert that fired eleven times and produced no action is either mis-tuned or should be a ticket, and deleting it is a real improvement rather than a loss of coverage. Alert sets grow monotonically unless something deliberately shrinks them, and the eventual state of an unpruned set is that on-call ignores all of it.

Finally, alert on the pipeline itself. If your metrics stop arriving, every rule above evaluates against no data and quietly reports that everything is fine — the failure mode where the dashboard is green because nothing is being measured. A staleness check on the request counter, and an alert when it drops to zero during hours it never should, is the cheapest insurance on this page.