Designing an AI Feature That Degrades Gracefully

Most AI features have exactly two states in the code and three in the designer’s head: working, and a spinner that never resolves. Degradation is the design of the states in between, and it is easier to get right if you write them down as an ordered ladder before you write the call.

A ladder, not a catch block

A try/catch around the model call gives you two outcomes: the good one and “something went wrong”. That is not enough resolution, because the failures are not alike. A model that is slow wants a different response from a model that returned a policy refusal, which wants a different response again from an account that has run out of credit. Collapsing all of them into one error state throws away the information that would have let you serve something useful.

The alternative is a ladder: a list of ways to answer the request, ordered from best to worst, each one cheaper and more certain than the one above it, ending in a rung that cannot fail. You climb down until something answers. The bottom rung is the important one, because its existence is what turns “the feature is broken” into “the feature is plain today”.

The five rungs

The specific ladder depends on the feature, but the shape recurs. From the top:

• Preferred model, full prompt. The thing you designed. Everything below this is a compromise you have decided in advance is better than nothing.
• Cheaper or faster model, same prompt. Costs quality in a way that is hard to see from the outside, which is exactly why this rung has to be recorded rather than silently taken.
• Cached or precomputed answer. A previous answer to the same question, possibly stale. For summaries, descriptions and classifications this is often nearly as good as fresh.
• Deterministic substitute. The non-AI implementation: keyword search instead of semantic search, the first two sentences instead of a summary, the rules engine instead of the classifier. Usually worse and always available.
• Honest absence. The feature is not offered, the rest of the page works, and the user is told in one sentence. This rung cannot fail, which is the only reason the ladder terminates.

Not every feature can supply all five. A feature that cannot supply rungs four or five is one whose entire value is the model, and that is worth knowing at design time rather than during an incident — it means the availability of the feature is exactly the availability of the dependency, and you should say so to whoever set the target.

The ladder in code

Written out, the pattern is short. The parts that matter are that each rung carries its own deadline, that the ladder stops descending when the overall budget is exhausted rather than trying every rung, and that the result names the rung that produced it.

type Rung<T> = {
  name: string;
  budgetMs: number;
  run: (signal: AbortSignal) => Promise<T>;
};

async function descend<T>(rungs: Rung<T>[], totalMs: number) {
  const deadline = Date.now() + totalMs;
  const errors: { rung: string; error: unknown }[] = [];

  for (const rung of rungs) {
    const remaining = deadline - Date.now();
    // No point starting a rung we cannot finish: that is dead work,
    // and for a metered dependency it is dead work you pay for.
    if (remaining < rung.budgetMs) continue;

    const ac = new AbortController();
    const timer = setTimeout(() => ac.abort(), rung.budgetMs);
    try {
      return { value: await rung.run(ac.signal), rung: rung.name, errors };
    } catch (error) {
      errors.push({ rung: rung.name, error });
    } finally {
      clearTimeout(timer);
    }
  }
  throw new Error("ladder exhausted");
}

The continue on an insufficient budget is the line people leave out, and it is the one that stops a degradation ladder from becoming a latency amplifier. Three rungs at eight seconds each is a twenty-four-second worst case, and a user who waited twenty-four seconds for the deterministic fallback would have preferred it immediately. Give the last rung a budget near zero so it is always reachable.

What the interface does on each rung

The interface question is not “how do we show an error”. It is what the user is able to do next, and the answer differs by rung.

Two failure modes deserve their own treatment rather than a rung. A policy refusal is not an outage — descending the ladder will usually produce another refusal, and it should be surfaced as a refusal. Exhausted credit is not an outage either; it is a billing state, and quietly serving rung four while the invoice problem goes unnoticed is how a degraded feature becomes permanent.

Record the rung, or you will never know

The dangerous property of a good ladder is that it hides the thing it is protecting you from. If rung two is decent, a provider being down for six hours looks like a normal day with slightly worse output. Nobody pages anybody, and the first sign is a support ticket about quality three weeks later.

So the rung is not a debugging detail — it is a metric. Emit it on every response, alert on the rate of descent rather than on individual failures, and put the distribution somewhere a human sees weekly. “Four per cent of answers came from rung three yesterday” is an actionable sentence. “No errors” is not, because a ladder that works produces no errors by construction.