When to Ship an AI Feature Behind a Flag

Every team already knows how to put a feature behind a flag. What is different here is that the thing most likely to need changing at three in the morning is not whether the feature is on — it is which model it calls, which prompt it uses, and how much it is allowed to do without asking.

Why the usual flag is not enough

A conventional feature flag answers one question with a boolean, and it is the right shape because a conventional feature has one failure mode: it is broken. An AI feature has several, and they want different responses.

The provider is degraded — you want a different model, not the feature off. A prompt change regressed quality — you want the previous prompt, which is not a code deploy. The feature is fine but a specific customer’s data is producing bad output — you want it off for them and on for everyone else. Spend is running above forecast — you want the cheap model or the degraded path, not an outage. A single boolean answers none of these, so the response to each becomes a deploy, and a deploy is the slowest tool available at the moment you most need speed.

There is a second reason, specific to this dependency. The behaviour you are flagging can change without you deploying anything, because the model is somebody else’s and it can be updated underneath you. Flags are usually a mechanism for controlling your own changes; here they are also the mechanism for reacting to changes you did not make, which is why detecting a provider-side behaviour change and having a flag to respond with are two halves of one control.

Four things to flag separately

Keeping these independent matters because they are pulled by different people for different reasons. An engineer switches models during a provider incident; a domain owner rolls back a prompt; a support lead disables a tenant; a manager lowers autonomy after a bad week. If all four are one switch, every one of those actions turns the feature off for everybody, and the flag stops being used because its cost is too high.

One implementation note that pays for itself: the resolved value of all four belongs in the log line for every request, next to the model id and the prompt version. Otherwise you have made behaviour configurable and simultaneously made it impossible to reconstruct what behaviour a given request got.

Rollout criteria

A flag with no plan for opening it becomes permanent at one per cent. Write the ramp down before the first user sees the feature; four points, each with an exit condition.

• Internal only, no time limit. Exit when the team has used it on real work rather than test input. The purpose is to find the failure modes that only appear on genuine data, and it is free.
• A small named cohort. Not a random percentage — people you can contact. Exit on qualitative feedback plus the absence of the failure classes you were worried about. Random percentages are for later, when you need statistics rather than explanations.
• A percentage ramp with a holdout. Now the statistics matter. Keep a holdout group that never gets the feature, because it is the only way to attribute a change in a business metric to this feature rather than to the season. Exit on the quality, cost and latency metrics you named in advance — the three axes together, since improving one at the expense of the others is the standard way an AI feature looks successful and is not.
• Default on, flag retained. Exit — that is, delete the flag — only after a period with no rollbacks and with the replacement control in place. Which, for the model and prompt axes, may be never; those two are configuration rather than a temporary rollout mechanism.

The criterion that is easiest to skip and hardest to recover is the holdout. Without it, six months later nobody can say whether the feature helped, and the argument about whether to keep maintaining it is settled by whoever is most confident.

The kill switch

A kill switch is not the same control as the flag, and conflating them is why kill switches fail when used. It is defined by four requirements, all of which are about what it must not need.

• No deploy. If pulling it requires a build, it is not a kill switch. This is the requirement everything else follows from.
• No engineer. The person who notices at 2am is on support. If only the author can pull it, the response time is however long it takes to wake them.
• No dependency on the thing being killed. A switch whose evaluation calls the provider, or whose configuration is fetched through the failing path, fails exactly when it is needed. Fail-safe defaults, cached locally, evaluated without a network call.
• No error state. Pulling it should produce the degraded feature, not a stack trace. The switch turns the model off, and something else must still answer — which means the degradation path has to exist before the switch is worth having.

And it has to be exercised. A kill switch that has never been pulled in production is a hypothesis. Pull it deliberately during a quiet hour, watch what the feature does, and confirm the degraded path is the one you designed rather than a spinner. Doing this once tends to find at least one thing, most often that the degraded path was never wired up on one of the call sites.

Retiring flags before they rot

Flags accumulate, and a stale one is worse than no flag: it is an untested code path that somebody will eventually enable. Two habits keep the set honest.

First, give every rollout flag an expiry date at creation — a date by which it is either deleted or explicitly converted into permanent configuration. The distinction is the useful part. A rollout flag is temporary scaffolding and should die; a model-selection or autonomy flag is a permanent control and should be documented, owned and exercised. Mixing the two categories is how a codebase ends up with forty flags nobody dares remove.

Second, delete the dead branch when the flag goes. A flag that is permanently true with the false branch still present is a path that is no longer tested and no longer true — and in an AI feature, the false branch is usually the deterministic fallback, which is precisely the code you will need during the next incident. Either keep it exercised or admit it is gone; leaving it unexecuted for a year and trusting it in an emergency is the worst of the three options.