Designing the Off-Switch for an AI Feature

“Put it behind a feature flag” is treated as one decision. For an AI feature it is four, with four different blast radii, and the incident that needs the narrow one is not the incident that needs the wide one. Reaching for the only switch you built is how a bad fifteen minutes becomes a bad afternoon.

One boolean is not an off-switch

The failures that require intervention on an AI feature are not one thing. A provider outage, a quality regression on one customer’s traffic, a runaway cost, and a safety incident all demand different responses, and a single feature boolean answers only the last of them — expensively, by removing the feature from everyone.

The four layers below are ordered from narrowest to widest. Build them in that order, and reach for them in that order too.

The four layers

Layer 3 is the one most often missing and the one most often wanted. A quality regression almost never affects all traffic equally, and a global switch turns a problem affecting some users into an outage affecting all of them.

Every layer must work without a deploy

A switch that requires a release is a switch whose response time is your release time, and release time is exactly what degrades when a system is in trouble. Three properties make a switch real:

1. It is read per request, not at start-up. A value read once at boot needs a restart to change, which is a deploy with extra steps.
2. It is stored where it survives a restart. A value in process memory is reset by every deploy, every crash and every scale event, and is enforced inconsistently across workers.
3. Flipping it is auditable. Who, when, and back to what. A switch nobody can explain the state of is a source of incidents rather than a remedy for them.

Per-request evaluation has a second benefit worth designing for: it means the reversal is also instant. A switch that takes effect immediately in one direction and requires a rebuild in the other is not a rollback path.

Which way the default fails

The most consequential decision is what happens when the switch cannot be read — the configuration store is unreachable, the row is missing, the value is malformed.

The general rule: pick the failure direction that is recoverable, and add a loud check for it. Both directions will occur; only one of them can be undone.

What the configuration looks like

The four layers are one record per feature, not four unrelated booleans, because reading them together is what lets a single lookup answer “what is this feature allowed to do for this caller right now”.

{
  "feature": "document_summary",
  "enabled": true,                  // layer 4 — off for everyone
  "route": "primary",               // layer 1 — named routes, not model ids
  "mode": "full",                   // layer 2 — full | degraded | template
  "disabled_for": {                 // layer 3 — narrow first, always
    "accounts": ["acct_71f", "acct_9c2"],
    "plans": [],
    "regions": []
  },
  "max_cost_micros_per_request": 40000,
  "updated_by": "…", "updated_at": "…", "previous": { … }
}

RESOLUTION, evaluated per request:
  if config is unreadable            -> OFF     (fail closed)
  if not enabled                     -> OFF
  if caller matches disabled_for     -> OFF
  else                               -> serve with (route, mode) and the cap

Three details in that shape carry most of the value. route names a route rather than a model, so switching provider is a configuration change and the application never learns a vendor’s model string. previous makes reverting a copy rather than a recollection, which matters at the moment somebody is flipping a switch under pressure. And the cost cap lives in the same record, because a runaway is one of the incidents this record exists to answer and it should not require a different system.

Resolution must be a pure function of the record and the caller. If it reads anything else — a second store, an environment variable, a cached bundle — then two callers can disagree about the state, and a switch whose effect you cannot predict is not one you will reach for.

Rehearsing the rollback

An untested rollback path is a hypothesis. Four things to verify before the feature launches, not after:

• Flip each layer in production and time it. If layer 4 takes eleven minutes, that is your worst-case exposure, and you should know the number before the incident rather than during it.
• Check what the user sees in the off state. Every layer needs a defined visible state. The most common defect here is a blank panel or a spinner that never resolves, because the off path was never rendered by anyone.
• Check what in-flight work does. Requests already running when the switch flips must either complete or abort cleanly. For an agent with side effects this is the hard part — a run halted between a tool call and its record is the worst possible state, which is an argument for making the run a persisted state machine.
• Check that it comes back. Flip it off and on. Cached state, warmed connections and stale client bundles all make the return path different from the outbound one.

What a flag cannot roll back

The honest limit, and the reason flags are necessary but not sufficient.

• Anything already sent. Emails, webhooks, messages, published pages. A switch stops the next one; it does not recall the last one. Features that emit externally need a review step or a delay window, not just a flag.
• Anything already written. Rows created, records updated, files deleted by a tool call. Rolling back the feature does not roll back its effects, and reconciling them afterwards is a separate project you should design before launch.
• Money already spent. A cost runaway caught by a switch still cost what it cost. The switch limits the duration, not the rate, which is why a hard budget belongs in the request path as well — see denial of wallet.
• Trust. If the failure was visible to customers, the switch ends the incident and not the consequence. That belongs in the write-up, not in the flag.

The wider mechanics of flagging model changes specifically — per-model flags, canaries, ramps — are in model feature flags and canary releases.