Feature Flags for Models and Prompts

Every mitigation in an AI incident runbook is a configuration change: pin the model, roll back the prompt, degrade the feature, cap the spend. Which means the flag system is not a nice-to-have around the edges of the feature — it is the mechanism by which the feature can be controlled at all.

Four kinds of flag

They have different lifetimes, different owners and different removal rules, and mixing them into one undifferentiated pile of booleans is how a codebase ends up with two hundred flags nobody dares delete.

Degradation is a ladder, not a switch

The default kill switch turns the feature off and shows an error. That is the worst available option: the user gets nothing, and you get a support ticket for every affected session. Model-backed features are unusually well suited to graded degradation, because there is almost always a cheaper, dumber, more reliable thing you could do instead.

export type ServiceLevel =
  | "full"        // primary model, full context, all tools
  | "reduced"     // smaller/cheaper model, trimmed context, no optional tools
  | "cached"      // serve a semantically similar cached answer if one exists
  | "static"      // deterministic non-AI path: templates, search, rules
  | "off";        // honest message; the rest of the product still works

// Chosen by whichever constraint binds first. Order is deliberate:
// spend and provider health outrank the operator's setting, because both
// can change faster than a human can react.
export function serviceLevel(s: Signals): ServiceLevel {
  if (s.hourlySpendUsd > s.spendCapUsd)          return "cached";
  if (s.allProvidersFailing)                     return "static";
  if (s.errorRate5m > 0.25)                      return "reduced";
  if (s.operatorLevel)                           return s.operatorLevel;
  return "full";
}

The static rung is the one teams skip and the one that pays off. Most AI features replaced something — a search box, a template, a rules engine — and that thing usually still exists three commits deep in the repository. Keeping it wired up as a fallback converts a total outage into a worse-but-working product.

Whatever the rung, tell the user. A degraded answer presented as a normal one is worse than an error, because it is acted on. A single line — this response used a faster model while we resolve an issue — costs nothing and prevents the support ticket that says the product got stupid.

Choosing the rung automatically rather than waiting for an operator is what makes the ladder worth building. The three conditions in that function — spend over cap, all providers failing, error rate elevated — are all things that can develop faster than a human responds, and all three have an obviously correct degradation. Leave the operator override in place for everything else, and put it below the automatic rules so that a forgotten manual setting cannot prevent the system from protecting itself.

The rungs also need to be exercised, because a fallback path that is never taken is a path that has silently stopped compiling. Route a small fixed percentage of traffic through reduced permanently, or run the static path in a synthetic check. Either way the degradation is being tested continuously rather than for the first time during an incident, which is when its bugs are least welcome.

Resolution that cannot fail closed

A flag system on the critical path of every inference call is a new dependency, and it must be a weaker one than the thing it protects. Three requirements, all of which are about the failure case:

• Never a synchronous network call. Flags are polled or streamed into memory in the background. The request path reads a local map. A flag service outage is invisible.
• Defaults compiled into the binary. A cold start with an unreachable flag service must still produce sane values. That means defaults live in code, not only in the flag service.
• Last known good on failure. If a poll fails, keep the previous map — do not fall back to defaults, which would undo an operator’s deliberate mitigation at the worst moment.

On the interface: OpenFeature is a CNCF specification for a vendor-neutral flag evaluation API with pluggable providers, and it is worth adopting even if you start with a hand-rolled provider, for the same reason as any interface — the SDK you choose today is not the one you will have in three years, and the call sites should not care.

There is a corollary about where flags are evaluated. Resolve them server-side, on the same process that makes the model call, and pass the resolved value down rather than re-resolving at each layer. A flag evaluated twice within one request can return two different answers if the background poll landed between them, and the resulting bug — a request that used one model’s parameters with another model’s prompt — is genuinely difficult to reason about after the fact.

Log the resolved variant

A flag whose resolved value is not recorded on the request makes every later analysis ambiguous: you cannot tell which requests got which treatment, so you cannot compare them. OpenTelemetry has semantic conventions for this — a feature flag evaluation is recorded with feature_flag.key, the resulting variant (named feature_flag.result.variant in current releases, and feature_flag.variant in older ones — another instance of the rename problem worth pinning), the provider name, and a context id.

In practice, put the small set of flags that affect model behaviour directly onto the request row as columns, and leave the long tail as span events:

alter table llm_request
  add column service_level  text,     -- full | reduced | cached | static
  add column model_variant  text,     -- which flag chose this model
  add column experiment_id  text;     -- null outside a rollout

-- What did each service level cost us, and did it work?
select service_level,
       count(*)                                                as n,
       round(avg(cost_usd), 6)                                 as avg_cost,
       round(avg(duration_ms))                                 as avg_ms,
       round(avg((not schema_valid)::int)::numeric, 4)         as schema_fail,
       round(avg(regenerated::int)::numeric, 4)                as regen
from llm_request
where environment = 'prod' and started_at > now() - interval '30 days'
group by 1 order by n desc;

That query answers a question worth knowing before an incident: how much worse is the reduced rung, actually? If regeneration rate barely moves, it is a candidate for the default and a cost saving. If it doubles, you know what degradation costs you and can decide accordingly.

Recording the resolved variant also settles an argument that otherwise recurs indefinitely. When someone reports that the product was worse last Tuesday, the flag columns turn a debate into a filter: those users were on reduced for two hours because the spend cap had engaged, and here is the count. Without the column, the same conversation has no evidence on either side and is decided by whoever is most confident.

Flag hygiene

• Every release flag has an owner and an expiry. Record both on creation. A stale release flag is dead code with a runtime branch and a false sense of control.
• Exercise the kill switches on a schedule. A switch that has never been flipped in production is a hypothesis. Flip each one in staging monthly and at least once in production during a quiet window.
• Audit changes to operational flags. Who, when, what, and ideally why. This is the timeline you will want during the postmortem, and it is the flag change that is most often the cause.
• Keep model choice out of entitlement logic. If the flag that pins a model during an incident also determines what a customer paid for, the incident mitigation has a billing consequence.
• Cap the number of interacting flags. Three flags with three values each is 27 configurations, and you have tested one. Prefer a single enumerated ServiceLevel over a combination of independent booleans.