Change Management for AI Rollouts

Rolling out an AI feature is usually described as a communication exercise. It is better run as a sequence of gates, because the things that actually stop a rollout — support load, cost per active user, a failure class nobody had seen — are quantities, and quantities can have thresholds.

What a rollout is actually gating

Three quantities grow with the number of people using the thing, and each has a level at which the rollout must pause regardless of how enthusiastic everyone is.

• Support load. Every new user generates questions, and the rate per user is roughly constant early on. If one user in twenty asks something in a given week, a 40-person stage produces two questions a week and a 1,200-person stage produces sixty — which is somebody’s job, and if that job is unassigned the answer becomes “the tool is confusing”.
• Cost per active user. Not per licensed user. The ratio between them changes as you widen: early cohorts are heavy users, later ones are light, so cost per active user often rises while cost per seat falls. Both numbers are needed and only one of them is usually tracked.
• Novel failure classes. Each new group brings inputs the previous groups did not have. The rate of genuinely new failure types — not new instances of known ones — is the best available signal that you have seen the distribution.

A rollout plan that does not name thresholds for those three is a schedule, not a plan, and schedules do not survive the first bad week.

The five stages

Five, and the boundaries are chosen so that each stage answers a question the previous one could not.

Stage 4 is worth resisting. Removing the manual path converts every provider outage into a work stoppage, and the saving from removing it is usually small compared with the cost of the first bad afternoon.

Exit criteria, as numbers

A stage ends when its criteria are met, not when a date arrives. Fill in your own thresholds; the shape is what transfers.

STAGE 0 -> 1
  failure taxonomy written, with an example of each class
  a rollback switch exists and has been used once on purpose
  cost per request measured, not estimated

STAGE 1 -> 2
  primary metric moved by at least the pre-registered effect
  no guardrail metric moved against us
  support questions per user per week  <  your threshold
  zero unresolved incidents of the "wrong user's data" class

STAGE 2 -> 3
  >= X% of the department turned it on unprompted
  of those, >= Y% still using it four weeks later
  cost per active user stable or falling for three weeks
  no new failure class for two consecutive weeks

STAGE 3 -> 4  (usually: do not)
  a degradation path exists and has been exercised
  the manual process can still be restored within one day

The four-week retention line in stage 2 is the one that does the most work, because first-week usage measures curiosity. A feature that everyone tries and a third keep is a real feature; one that everyone tries and nobody keeps has told you something important that a satisfaction survey would have hidden.

Two rules keep the gates honest. A stage may be repeated but not skipped: if stage 2 fails its criteria, the answer is to fix something and run stage 2 again, not to proceed on the grounds that the next cohort might be different. And criteria are set before the stage starts, for the same reason a pilot’s success threshold is — a number chosen after seeing the result is not a gate, it is a summary. Write the thresholds for all four transitions at the beginning, when nobody yet has an interest in where they land.

Why opt-in has to come before default-on

This ordering is not politeness. It is the only way to separate two explanations that look identical from the outside: the tool is not good enough and the change is unwelcome.

In an opt-in stage, the people using it chose to. If they stop, the tool is the problem, because nothing else changed for them. In a default-on stage, non-use is ambiguous — it could be quality, habit, workload, or a manager who told the team not to bother. Running opt-in first buys you an unambiguous quality signal that you cannot get later, and it costs a few weeks.

There is a second, more practical reason. An opt-in cohort self-selects for people who will tell you what is wrong with it. That is a stream of free failure reports arriving before the feature is in front of anyone who will be annoyed by them, and it is worth more than the same reports arriving as tickets a month later.

The part people call trust

The received wisdom is that adoption depends on trust, which is true and useless because trust is not something you can build on a Tuesday. What you can build are the mechanical properties that make trust the rational response — and they are all design decisions with a clear shape.

• Reversible before consequential. Draft rather than send. Suggest rather than apply. Propose rather than commit. A user who can inspect and edit before anything happens is not taking a risk and does not need to trust anything.
• Show the source. An answer with a citation the reader can open is checkable in seconds; the same answer without one has to be believed or redone. This is the practical case for grounding as an architecture decision and for citations a user can actually verify.
• Let it say it does not know. A system that abstains rather than guessing is worth more to an expert user than one that is right slightly more often and never signals doubt, because the second one has to be checked every time.
• Make the off switch visible. A user who can see how to turn it off usually does not, and one who cannot see how spends the first week looking for it.
• Answer questions where the work happens. A named person in the team’s own channel, not a training portal. This is the support-load line from the first section, staffed deliberately rather than absorbed by whoever is nearest.

None of that is culture work. It is a list of product decisions and a staffing decision, and it is the difference between a rollout that needs constant advocacy and one that proceeds on its own — which is also, conveniently, what makes the measurement at the end mean anything.