On-Call Runbooks for AI Services

A runbook is a decision tree for somebody who was asleep four minutes ago. It is not documentation, not an architecture overview and not a list of possible causes: it is symptom, first command, what the output means, and what to do about each answer. Below are five, for the five alerts that actually fire on an AI service.

What a runbook is, and is not

The format that works has five parts, and pages that omit the third are the ones nobody uses.

Two rules for the whole set. Every command must be safe to run read-only, or be clearly marked as changing something. And every runbook opens with mitigation before diagnosis, because restoring service and understanding the cause are different jobs and doing them in the wrong order costs users.

Runbook 1: latency spike

SYMPTOM  InferenceQueueWaitHigh — p95 queue wait > 10 s for 10 min.
         Users report slow or hanging responses.

FIRST    Is this load, or is it capacity?
COMMAND
    kubectl get hpa llm-server
    kubectl get pods -l app=llm-server -o wide
    # and the two panels: request rate, and ready replicas

DECISION
    Request rate up >30% vs same hour last week  → LOAD.        Go to A.
    Request rate flat, ready replicas down       → CAPACITY.    Go to B.
    Request rate flat, replicas at desired count → DEGRADATION. Go to C.
    HPA shows desired > current for >5 min       → SCHEDULING.  Go to D.

A. LOAD
   1. Confirm the HPA is scaling: kubectl describe hpa llm-server
   2. If at maxReplicas, raise it — this is a config change, needs approval:
        kubectl patch hpa llm-server --type=merge \
          -p '{"spec":{"maxReplicas":20}}'
      Only helps if GPU capacity exists. Check node allocatable first:
        kubectl describe nodes | grep -A4 "Allocatable" | grep nvidia.com/gpu
   3. If no spare devices: shed load rather than queue it. Lower the free-tier
      admission bound so paying traffic is protected.
   4. Tell support what is degraded and for whom.

B. CAPACITY (replicas missing)
   1. kubectl get pods -l app=llm-server
   2. Pending  → kubectl describe pod <name> | tail -20
        "Insufficient nvidia.com/gpu" → no free devices; check the device
        plugin DaemonSet is Running on every GPU node:
          kubectl get ds -n kube-system nvidia-device-plugin-daemonset
        A crashed plugin makes healthy hardware invisible. Restart it.
   3. CrashLoopBackOff → kubectl logs <name> --previous
        OOMKilled → host memory limit, not GPU. Raise or reduce concurrency.
        CUDA error → this is Runbook 4.
   4. Terminating for >2 min → a pod is not honouring SIGTERM; it will be
      killed at the grace period. Note it as a follow-up, do not wait.

C. DEGRADATION (right number of pods, still slow)
   1. Are the GPUs throttled?
        kubectl exec <pod> -- nvidia-smi \
          --query-gpu=clocks_throttle_reasons.active,temperature.gpu,\
power.draw,power.limit --format=csv
      Throttling → Runbook 4, thermal branch.
   2. Has the input distribution changed? Compare p50 input tokens and p50
      output tokens with last week. A prompt that grew 4x costs 4x prefill,
      and this is a surprisingly common cause of a "sudden" slowdown.
   3. Was there a deploy? Check deploy markers on the dashboard.
        kubectl rollout history deployment/llm-server
      If a model version changed in the last hour → Runbook 5.
   4. Is a dependency slow? Retrieval, embeddings, a database. Look at the
      trace for one slow request end to end before guessing.

D. SCHEDULING (HPA wants more pods, none appear)
   1. kubectl get events --sort-by=.lastTimestamp | tail -30
   2. Cluster autoscaler unable to add nodes → quota or capacity. Neither is
      fixable in the next ten minutes. Go to A.3 and shed load.

MITIGATE FIRST, ALWAYS
   If users are affected and the cause is not clear within 10 minutes:
   shed low-priority traffic, or fail over to the secondary provider —
   both are reversible, and both buy time to diagnose.

ESCALATE  after 20 minutes without a branch identified, or immediately if
          error rate exceeds 5%. Hand over: which branch you ruled out,
          the commands you ran, the trace id of one slow request.

Runbook 2: 429 storm from a provider

SYMPTOM  Provider returning 429 at high rate. User-visible failures or
         very long latencies from retry backoff.

FIRST    Whose limit, and which one?
COMMAND
    # Group recent provider errors by key fingerprint and limit type
    logcli query '{app="llm-gateway"} |= "429"' --since=15m \
      | jq -r '[.key_fp, .limit_type, .model] | @tsv' | sort | uniq -c | sort -rn

DECISION
    Concentrated on one key/project  → your account limit.       Go to A.
    Spread across all keys           → provider-wide event.      Go to B.
    Only one model identifier        → per-model limit.          Go to C.
    Started exactly at a deploy      → your own traffic changed. Go to D.

A. YOUR ACCOUNT LIMIT
   1. Read the response headers you logged: remaining requests, remaining
      tokens, reset time. Provider header names differ — check yours.
   2. Requests-per-minute limit → reduce concurrency now.
      Tokens-per-minute limit  → reduce max_tokens or batch size; concurrency
      alone will not help, because the constraint is tokens.
   3. Apply backpressure at admission rather than retrying harder. Retrying a
      429 without reducing the offered rate makes it worse and is the single
      most common mistake here.
   4. Request a limit increase. This has a lead time; it is not a fix now.

B. PROVIDER-WIDE EVENT
   1. Check the provider status page. Note the time you checked.
   2. Fail over to the secondary provider or model:
        kubectl set env deployment/llm-gateway PRIMARY_ROUTE=secondary
   3. Verify output still validates — a fallback model may not honour the
      same schema. If it does not, prefer degrading the feature to returning
      malformed data downstream.
   4. Set a reminder to fail back, and check cost: the fallback may be
      more expensive per token.

C. PER-MODEL LIMIT
   1. Route that model's traffic to an alternative for now.
   2. Check whether one caller is responsible:
        group the 429s by X-Cost-Service. One service that started a backfill
        is the usual answer, and pausing it fixes this in one action.

D. YOUR OWN TRAFFIC CHANGED
   1. kubectl rollout history deployment/<caller>
   2. Look for: a new retry loop, a removed cache, a prompt that grew, a
      batch job that started. Compare requests-per-minute per service against
      yesterday.
   3. Roll back the caller, not the gateway.

ESCALATE  if failover is unavailable or fails, or if the provider event
          exceeds 30 minutes with no status update. Hand over: which branch,
          the current offered rate, and whether the fallback is validating.

Runbook 3: cost spike

SYMPTOM  Spend alert: today's spend is >2x the trailing 7-day average
         for this hour of the week.

FIRST    Volume, or cost per request?
COMMAND
    # From the request log: requests, tokens and cost per hour, last 24h
    SELECT date_trunc('hour', ts) AS h,
           count(*)                          AS requests,
           sum(input_tokens)                 AS in_tok,
           sum(output_tokens)                AS out_tok,
           sum(cost_usd_micros)/1e6          AS cost,
           sum(cost_usd_micros)/nullif(count(*),0)/1e6 AS cost_per_request
    FROM model_calls
    WHERE ts > now() - interval '24 hours'
    GROUP BY 1 ORDER BY 1;

DECISION
    requests up, cost_per_request flat     → VOLUME.       Go to A.
    requests flat, cost_per_request up     → UNIT COST.    Go to B.
    both up                                → both; do A first.
    a single tenant dominates              → ABUSE or bug. Go to C.

A. VOLUME
   1. Group by service, feature and tenant for the spike window. One of them
      will be nearly all of it.
   2. Legitimate growth        → capacity and budget conversation, not an
                                 incident. Close the alert with a note.
   3. A backfill or batch job  → is it meant to be running now? Pause it.
   4. A retry loop             → count attempts per request_id. If the p99 is
                                 above your retry budget, you have a loop.
                                 This is the expensive one; stop it first.

B. UNIT COST
   1. Did the model change? Compare model identifier distribution to
      yesterday. A silent provider-side change or a routing change can move
      cost per request with no code deploy.
   2. Did the prompt grow? Compare p50 input_tokens to last week. A retrieval
      change that returns 20 chunks instead of 5 quadruples prefill cost.
   3. Did caching stop working? Compare cached_input_tokens to total input
      tokens. A cache hit ratio that fell to zero usually means a prefix
      changed — a timestamp or a session id at the start of the prompt.
   4. Did output length grow? A prompt edit that removed a length instruction
      is a common and easily missed cause.

C. SINGLE TENANT DOMINATES
   1. Is it a paying customer within their plan? Then this is capacity.
   2. Is it a free-tier account? Check for credential abuse — spend from a
      stolen key looks exactly like this.
   3. Apply the per-tenant cap immediately; investigate afterwards.

IMMEDIATE MITIGATION, if spend is still climbing:
   Lower the global spend cap so the system refuses rather than bills:
     kubectl set env deployment/llm-gateway DAILY_BUDGET_USD=<current+buffer>
   Refusing requests is recoverable. An unbounded bill is not.

ESCALATE  to the budget owner immediately if projected daily spend exceeds
          the monthly budget divided by the days remaining.

The mitigation line is the important one, and it is worth having a cap that can be lowered in one command before you need it. Budget controls covers the design, and denial of wallet covers the case where the spike is deliberate.

Runbook 4: GPU device fault

SYMPTOM  GPUDeviceFault — XID error or uncorrected ECC. Or: pods on one
         node crashing with CUDA errors while other nodes are healthy.

FIRST    What does the device say?
COMMAND
    # On the node (or via a debug pod with host access):
    nvidia-smi -q | grep -Ei 'xid|ecc|retired|remap|throttle'
    dmesg -T | grep -i -E 'xid|nvrm|nvidia' | tail -40

DECISION
    XID present                       → hardware or driver.  Go to A.
    Uncorrected ECC / retired pages   → failing memory.      Go to A.
    No XID, thermal throttle active   → cooling.             Go to B.
    No XID, "out of memory" in app    → your workload.       Go to C.
    Device missing from nvidia-smi    → fell off the bus.    Go to A, urgent.

A. HARDWARE OR DRIVER
   1. Cordon and drain the node. This is a change; it is the right one.
        kubectl cordon <node>
        kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
   2. Record the XID number and the exact dmesg lines before doing anything
      else — a reboot destroys the evidence. Look the number up in NVIDIA's
      XID documentation; the classes differ from "application bug" to
      "replace the card".
   3. If it is a cloud instance: terminate it and let the pool replace it.
      Do not try to repair it. If it is your hardware: raise the RMA with
      the serial number and the XID.
   4. Confirm capacity is restored before closing:
        kubectl get nodes -l accelerator --no-headers | wc -l

B. THERMAL OR POWER
   1. nvidia-smi --query-gpu=temperature.gpu,power.draw,power.limit,\
        clocks_throttle_reasons.active --format=csv
   2. Thermal, one node       → cooling or a physical problem. Cordon it and
                                raise it with the operator.
   3. Thermal, whole rack     → facility issue. Escalate to the provider now;
                                this will not resolve on its own.
   4. Power cap               → the enforced limit is below the board's
                                capability. It may be deliberate. Note it as
                                lost capacity in the capacity review rather
                                than fighting it at 3am.

C. APPLICATION OUT OF MEMORY
   1. This is not a device fault. The card is fine; the workload asked for
      more memory than exists.
   2. Reduce max concurrent sequences and the KV cache limit in the server
      config, then restart the pod.
   3. Check whether a longer-context request arrived — KV cache grows with
      context length times batch size, so one long request can push a batch
      that fits at 8k over the limit at 32k.

ESCALATE  immediately for uncorrected ECC or a missing device: those are
          hardware faults and waiting does not improve them.

Runbook 5: quality regression after a deploy

SYMPTOM  Schema validation pass rate, tool-call success, or thumbs-down
         rate crossed its threshold. No errors, no latency change.

FIRST    What changed, and when exactly?
COMMAND
    # Align the metric change with the deploy timeline.
    kubectl rollout history deployment/llm-server
    git log --since="6 hours ago" --oneline -- prompts/ deploy/models.yaml
    # And: has the provider's model identifier resolved differently?
    #   compare the model version string recorded on responses, before/after

DECISION
    Change starts within 15 min of a deploy    → OUR CHANGE.   Go to A.
    No deploy, model version string changed    → THEIR CHANGE. Go to B.
    No deploy, no version change, gradual      → INPUT DRIFT.  Go to C.
    Confined to one tenant or one feature      → SCOPED.       Go to D.

A. OUR CHANGE
   1. Roll back first, diagnose second. If the deploy was a canary, set its
      traffic to zero — seconds, not a rollout:
        kubectl patch <route/vs/ingress> ... weight: 0     # your split mechanism
      If it was a full rollout:
        kubectl rollout undo deployment/llm-server
   2. Confirm the metric recovers within one window. If it does not, the
      deploy was not the cause — go back to the decision table.
   3. Capture 20 failing examples before the canary pods are removed. Without
      the inputs you cannot reproduce it, and they disappear with the pods.

B. THEIR CHANGE (provider updated the model)
   1. Pin to a dated or versioned model identifier if the provider offers one.
      This is the fix, and it is also the prevention.
   2. Run the eval set against both versions and quantify the difference
      before deciding whether to stay.
   3. If no pinned version exists, failover to an alternative model is the
      only lever. Check the output contract holds.

C. INPUT DRIFT
   1. Compare the input distribution with last week: language mix, length,
      document types, new customer onboarded.
   2. This is not an incident to be resolved tonight. Downgrade to a ticket,
      capture examples, and fix it in the prompt or the eval set.

D. SCOPED TO ONE TENANT OR FEATURE
   1. Almost always their data or their configuration, not your model.
   2. Check for a document format, language or size the pipeline mishandles.
   3. Disable the feature for that tenant if it is producing wrong answers.
      A feature that is off is better than one that is confidently wrong.

ESCALATE  to the owning team, not to infrastructure. This class of failure
          is rarely fixed by the person on call for the platform. Hand over:
          the metric, the exact change time, the 20 captured examples.

The rollback-first instruction in branch A is deliberate and worth stating as policy rather than as advice. Canary and blue-green deploys is what makes it a seconds-long operation instead of a rollout, and the abort criteria written there are what let an on-call engineer decide without a debate.

Keeping them true

A runbook rots faster than code because nothing fails when it is wrong. Four practices keep them honest.

• Link from the alert, always. Every alert carries a runbook annotation with a URL. If an alert has no runbook, either write one or delete the alert — an alert nobody knows how to action is a pager that trains people to ignore pagers.
• Edit during the incident, not after. The person using it has just discovered what is wrong with it. A one-line fix then beats a documentation task nobody picks up.
• Keep them in the repository. Same review, same history, changed in the pull request that changes the system. A runbook in a wiki drifts from the deployment it describes.
• Exercise them. The game-day procedure in chaos testing a provider outage is how you discover that the dashboard was renamed, the command needs a flag that no longer exists, and the escalation contact left in March.

One more thing worth writing down next to them: what you are not allowed to do. Cross-border failover where residency forbids it, disabling an audit log to reduce load, raising a spend cap without an approver. Those decisions are much harder at 3am than at any other time, and the runbook is where the answer should already be. Incident response for AI features covers the wider process these five sit inside.