SLOs for AI Features

The standard objection to SLOs for AI features is that you cannot put a number on “correct”. That is true and it is not an argument against SLOs — it is an argument for being careful about which properties go in the budget and which are tracked beside it.

An SLI is a precise sentence

Most SLO work fails at the definition, not the target. An indicator needs three things nailed down before a percentage means anything: which events count (the denominator), which of those count as good (the numerator), and where it is measured.

“99.9% availability” is not an SLI. This is:

SLI: assistant response success

  Valid events   HTTP requests to POST /v1/assistant/message
                 from authenticated users in production,
                 excluding requests rejected for quota (429) and
                 excluding requests cancelled by the client.

  Good events    Requests that returned 200, produced a first token
                 within 5s, and whose body validated against
                 AssistantResponse v3.

  Measured at    The application edge, from server-side request logs.
                 Not at the provider boundary — a failover that
                 succeeded is a good event even though one upstream
                 call failed.

  Objective      99.0% of valid events are good, over a rolling 30 days.

Note what the exclusions do. Quota rejections are a product decision, not a reliability failure, and leaving them in means a successful sales quarter burns your error budget. Client cancellations are the user’s choice. Both belong out of the denominator, and both are the kind of thing that is obvious in advance and infuriating to discover mid-incident.

“Measured at the edge” is the second load-bearing line. If your SLI counts provider calls, a retry that succeeded shows as a failure, and your budget burns for something no user experienced.

Split the contractual from the fuzzy

Divide the properties you care about into two lists, and only one of them gets an error budget.

Put the first list in the SLO and the error budget. Track the second on a dashboard with its interval drawn, and review it on a cadence rather than alerting on it. A budget that can be exhausted by resampling noise is a budget nobody will respect.

The pleasant surprise is how much of “wrong” the contractual list catches. An answer that omits the citation, silently truncates, drops a required field or replies in the wrong language is wrong in a way you can count exactly.

This also suggests a design direction rather than only a measurement one. Every property you can move from the fuzzy list to the contractual list makes the feature more measurable — asking for structured output instead of prose, requiring citations by schema rather than by instruction, constraining a classification to an enumerated set. Those are product decisions with real trade-offs, but they are worth weighing knowing that one side of the trade-off buys you an SLI you can actually enforce.

Error budget arithmetic

The budget is what makes an SLO a decision tool rather than a poster. At a 99.0% objective over 30 days with 1,000,000 valid events, the budget is 10,000 bad events. That number is the input to two concrete policies:

• Alerting. Page on burn rate against this budget rather than on raw thresholds — the multiwindow rules in the alerting page take the budget as their only parameter.
• Release policy. Budget remaining above a stated level, model migrations and prompt changes proceed. Below it, only reliability work ships. Writing that down before an incident is the entire mechanism by which an SLO changes anything.

Pick the objective from what the product needs, not from how many nines sound impressive. Three nines on a feature backed by third-party model APIs means roughly 43 minutes of budget across a month for everything — your bugs, their outages, every deploy — and the usual result of setting it there is that it is quietly ignored by the second month. An objective you will actually enforce at 99.0% is worth more than an aspirational 99.9%.

A quality objective that is not a lie

If you do want a quality objective, make it a statement about a measurement procedure rather than about truth. Something of this shape:

Objective: on a stratified random sample of 500 production responses
per week, graded by rubric R v2 with judge model M pinned to snapshot
M-2026-05-01, at least 90% score "acceptable" or better.

Reported as: the observed proportion with a 95% interval. At n = 500
and p ≈ 0.9, the interval half-width is 1.96·sqrt(0.9·0.1/500) ≈ 2.6pp,
so 90% is indistinguishable from anything in 87.4%–92.6%.

Breach requires: two consecutive weeks below the bound, not one.
Judge and rubric versions change only alongside a re-baseline.

Everything awkward about quality measurement is made explicit there: the sample size, the judge, its version, the interval, and the rule that one bad week is not a breach. Pinning the judge is not bureaucracy — an unpinned judge model is a measuring instrument that changes silently, and then you cannot tell whether the score moved or the ruler did.

Your provider’s SLA is not your SLO

Model providers publish availability commitments for enterprise tiers, and they are about their API returning responses. Yours is about your feature working. Three differences matter:

• Your SLI includes your own code. Retrieval, parsing, validation, tool execution and rendering are all inside your numerator and outside their SLA.
• Failover changes the arithmetic in your favour. If a request that failed on one provider succeeded on another within the latency budget, it is a good event. Measuring at the edge is what lets you claim that; measuring per upstream call is what stops you.
• Rate limits are not outages, and they are yours. Being throttled because you exceeded your own quota is a capacity failure on your side. Exclude quota rejections from the SLI if they are a product behaviour; include them if they are a surprise, and be clear which one you mean.

The practical implication is that the reliability work with the best return is usually not choosing a more reliable provider. It is having a second one configured, a validated fallback path, and a degradation mode that is better than an error page — all of which show up directly in the numerator.

One practical warning about publishing any of this externally. An SLO is an internal engineering target; an SLA is a contractual commitment with money attached. They should not be the same number, and the internal one should be tighter, because you want the budget to be exhausted and to trigger a change in behaviour well before a customer has a claim. Teams that publish their SLO as their SLA discover this the first time a provider outage lands, and the discovery is expensive.