Skip to content
Multigrid

Observability & LLMOps

How to see what your model calls are doing in production — what to log, how to trace it, what to alert on, and how to attribute the bill.

An LLM feature fails differently from the rest of your stack. The request returns 200, the latency is normal, the error rate is flat, and the answer is wrong. Nothing in a classical monitoring setup moves. Meanwhile the one signal that is moving — spend — moves in a dashboard owned by finance, six weeks after the deploy that caused it.

These pages are about closing that gap with ordinary engineering. There is a published specification for the trace shape (OpenTelemetry’s GenAI semantic conventions), the interesting fields are nameable, and most of the hard questions — which feature is spending the money, did the provider change the model under us, can I replay this request six hours later — reduce to having written down the right columns before you needed them.

Every page here ships an artefact rather than an exhortation: a schema, a query, a middleware, a config. Nothing on this cluster asks you to improve your visibility.

What to Log for Every LLM Call

A concrete column list for a request log, including the fields nobody adds until the first incident that needed them.

5 min read

Tracing an LLM Application With OpenTelemetry

The GenAI semantic conventions — span names, attribute names, metric names — and working instrumentation that emits them.

5 min read

Debugging a Bad Response Six Hours Later

What has to be in the trace for a complaint about yesterday's answer to become a reproducible request today.

5 min read

Cost Attribution: Which Feature Is Spending Your Budget

A metadata design that survives refactors, plus the queries that turn one invoice line into a per-feature breakdown.

6 min read

Per-Customer Cost Tracking in a Multi-Tenant App

How to aggregate model spend per tenant reliably enough to bill on it, including shared cost, late rows and corrections.

5 min read

Alerting on LLM Metrics Without Alarm Fatigue

Which LLM signals deserve to wake someone, expressed as level-triggered burn-rate alerts with concrete windows and multipliers.

5 min read

Detecting Quality Regressions in Production

The signals that indicate output quality has moved, and the arithmetic that tells you whether your traffic volume can detect the change you care about.

5 min read

Silent Model Updates: Detecting a Behaviour Change

How to tell whether the model behind your alias changed, using response-model diffing and a canary suite with an honest tolerance band.

5 min read

Prompt Registries and Deploying a Prompt Change

A versioned prompt store that decouples prompt releases from code releases, without making your inference path depend on a network call.

6 min read

Shadow Traffic: Testing a New Model on Real Requests

Mirroring production requests to a candidate model without touching the user's response, including the side-effect and PII problems.

6 min read

Canary Releases for Model Migrations

Percentage rollout for a model change, with sticky assignment, a stage ladder sized by statistics, and rollback triggers that fire without a human.

5 min read

Sampling: How Much Traffic Do You Need to Log?

Head versus tail sampling, the arithmetic of catching rare events, and a collector policy set that keeps errors at 100% while dropping the boring majority.

5 min read

PII in Your Logs: The Compliance Time Bomb

Why prompt logging accumulates personal data faster than any other log you own, and a capture-time redactor with its limits stated honestly.

5 min read

Building an LLM Cost Dashboard

Five charts, each shipped as SQL against a stated schema, with what each one is for and what it cannot tell you.

6 min read

SLOs for AI Features

How to write service level objectives for a feature whose main failure mode is being wrong, without pretending correctness is binary.

5 min read

Incident Response for AI Features

A runbook for the failure modes classical SRE does not have — the service is up, the answers are wrong, and nothing is red.

5 min read

Feature Flags for Models and Prompts

A flag taxonomy for model-backed features, with degradation as a ladder rather than an on/off switch, and evaluation that cannot fail closed.

5 min read

Replaying Production Traffic Against a New Model

Building a replay corpus from your request log, making the replay hermetic, and diffing two sets of outputs without a human reading all of them.

6 min read

Observability Vendors vs Rolling Your Own

What the categories of LLM observability tooling actually give you, what each one locks you into, and how to keep the exit cheap.

4 min read

The Metrics That Matter: A Minimal LLM Dashboard

Nine metrics, each defined precisely enough to implement — name, type, unit, labels, and what it deliberately excludes.

5 min read

Other topics

Observability & LLMOps · Multigrid