Circuit Breakers for Flaky AI Dependencies

The circuit breaker is a twenty-year-old pattern with well known settings, and nearly all of those settings were chosen for calls that take milliseconds and cost nothing. Point one at an inference endpoint without changing them and it will either never trip or never reset.

What a breaker buys you here

A breaker sits in front of a dependency, counts failures, and once there have been enough it stops calling for a while and fails immediately instead. Three things follow, and the third is the one specific to this dependency.

• Latency stops collapsing. When a provider is timing out, every request in flight is holding a worker, a connection and a slot in your concurrency limit for the full timeout. Failing fast returns all three immediately, which is what stops one sick dependency from taking down endpoints that never touch it.
• The fallback path gets used. A degradation ladder with an eight-second first rung is a ladder that adds eight seconds to every request during an outage. An open breaker skips rung one in microseconds, so degradation becomes cheap rather than merely possible.
• You stop buying failures. A timed-out call is often a call the provider completed and billed. During a bad ten minutes, a breaker is the difference between a hundred wasted generations and a handful of probes.

Why the default thresholds are wrong

The canonical configuration is something like “trip when 50% of requests fail within a 10-second rolling window, with a minimum of 20 requests”. Those numbers assume a service handling hundreds of calls per second, where 20 requests is a fraction of a second of traffic.

Now suppose a feature makes four model calls a minute, each taking five seconds. A 10-second window never contains 20 requests, so the minimum-volume guard is never satisfied and the breaker never trips — it will sit closed through an hour-long outage. The failure is not a bad threshold; it is the wrong unit. Time-based windows encode an assumption about arrival rate that does not hold.

Count requests instead. A ring buffer of the last N outcomes, with a trip condition expressed as “k of the last N”, behaves identically at any arrival rate. Pick N from how long you are willing to stay closed while broken: at your normal request rate, N calls is the detection delay you are accepting. Pick k from your tolerance for a false trip against a dependency whose baseline error rate is not zero — a majority of a window of twenty is a reasonable starting shape, and it is a starting shape rather than a recommendation because the right value depends on a baseline you have and this page does not.

Add a staleness rule to the buffer as well. Outcomes from an hour ago should not count towards a trip today, so discard entries older than a few minutes even though the window itself is counted in requests. This is the one place time re-enters, and it re-enters as an expiry rather than as the window.

Which errors trip it

A breaker exists to detect that the dependency is unwell. An error caused by your request is not evidence of that, and counting it will open the breaker for every other caller because one client sent malformed JSON.

An implementation

type Outcome = { ok: boolean; at: number };

export class Breaker {
  private ring: Outcome[] = [];
  private openedAt = 0;
  private probing = false;

  constructor(
    private readonly n = 20,          // window, counted in requests
    private readonly k = 11,          // failures in the window that trip it
    private readonly coolDownMs = 30_000,
    private readonly staleMs = 300_000,
  ) {}

  private state(): "closed" | "open" | "half" {
    if (!this.openedAt) return "closed";
    return Date.now() - this.openedAt >= this.coolDownMs ? "half" : "open";
  }

  async run<T>(call: () => Promise<T>): Promise<T> {
    const state = this.state();
    if (state === "open") throw new BreakerOpen(this.retryAfter());
    // In half-open, exactly one probe is allowed through. Everyone else is
    // rejected: a stampede of probes is how a recovering provider is knocked
    // back over, and here each probe is also a purchase.
    if (state === "half") {
      if (this.probing) throw new BreakerOpen(this.retryAfter());
      this.probing = true;
    }

    try {
      const value = await call();
      this.record(true, state);
      return value;
    } catch (error) {
      if (trips(error)) this.record(false, state);
      throw error;
    } finally {
      if (state === "half") this.probing = false;
    }
  }

  private record(ok: boolean, state: "closed" | "open" | "half") {
    if (state === "half") {
      // One good probe closes it; one bad probe restarts the clock.
      this.openedAt = ok ? 0 : Date.now();
      if (ok) this.ring = [];
      return;
    }
    const cutoff = Date.now() - this.staleMs;
    this.ring = [...this.ring, { ok, at: Date.now() }]
      .filter((o) => o.at >= cutoff)
      .slice(-this.n);

    const failures = this.ring.filter((o) => !o.ok).length;
    if (this.ring.length >= this.n && failures >= this.k) this.openedAt = Date.now();
  }

  private retryAfter() {
    return Math.max(0, this.openedAt + this.coolDownMs - Date.now());
  }
}

Notice that BreakerOpen carries a retry-after. An open breaker is not just a failure; it is a failure with a known expiry, and passing that upward lets a queue reschedule intelligently instead of burning its own retry budget against a door you know is shut.

The half-open probe costs money

In a classical breaker the half-open probe is free — a health check, a cheap read. Here the probe is a real generation with a real price, and that changes two decisions.

First, probe with the smallest possible request rather than replaying a user’s. A short prompt with a low max_tokens tells you the same thing about connectivity and authentication as a full call, for a fraction of the cost. If you can afford to probe with a real request, prefer one that would have been made anyway — but never probe with a request whose duplicate would be harmful, because a probe is by definition an attempt at something you believe may fail.

Second, back off the cool-down on repeated failure. A fixed thirty-second cool-down against an outage lasting an hour is a hundred and twenty probes. Doubling the cool-down on each failed probe, capped at a few minutes, turns that into a handful. The classical pattern usually omits this because the probe was free; here it is the difference between noticing recovery promptly and paying to ask a dead service the same question two thousand times.

One breaker per what?

A single global breaker is almost always wrong, because the failure domains are not global. The useful granularity is the smallest thing that can fail independently, which is typically the pair of provider and model: one model being withdrawn or overloaded says nothing about the others on the same account, and one provider’s region being down says nothing about a different provider serving the same model.

The cost of finer granularity is that each breaker sees fewer requests, so detection is slower. If you have many low-traffic model-provider pairs, keep the per-pair breakers but add a parent breaker at the provider level fed by all of them — the child detects a bad model, the parent detects a bad provider, and a failure at either level opens the right amount of the system. Whatever you choose, export the state as a metric with the pair as a label. An open breaker that nobody can see is indistinguishable from a dependency nobody is calling.