Skip to content
Multigrid

Building reliable AI applications

The distributed-systems patterns an AI feature needs — idempotency, deadlines, breakers, queues, degradation — and the parts of each that change when the dependency is slow, non-deterministic and billed per attempt.

A model call is a network call to a dependency you do not control. That much you have handled before. What is unfamiliar is the combination of properties: it takes seconds rather than milliseconds, so every timeout heuristic you have is off by two orders of magnitude. It is non-deterministic, so you cannot recover a lost result by running it again. It streams, so a failure can arrive after you have already sent the client a 200. And it is metered, so a retry is a purchase and a bug in your retry policy is a line on an invoice.

Every pattern in this cluster is older than the models. Circuit breakers, bulkheads, deadline propagation, claim-before-call idempotency, bounded queues, load shedding, compensating transactions — none of it was invented for AI. These pages assume you know roughly what each one is and spend their words on the part that changes: which threshold is wrong by default, which failure is silently billable, and which piece of the classical pattern assumes a cheap idempotent dependency and quietly stops being safe.

Designing an AI Feature That Degrades Gracefully

A degradation ladder for a feature whose most interesting dependency is also its least reliable one, and what the interface shows on each rung.

6 min read

Queues and Background Jobs for Slow AI Calls

When to move a model call off the request path, the job state machine that results, and the visibility-timeout hazard that makes an at-least-once queue bill you twice.

6 min read

Idempotency for Expensive Operations

Claim-before-call idempotency for an operation you are billed for, with the race condition written out and the schema that closes it.

7 min read

Timeouts and Deadlines Across a Multi-Step Pipeline

Why per-call timeouts do not compose, how to propagate an absolute deadline through a chain of model calls, and what to do when the remaining budget is too small to be useful.

7 min read

Circuit Breakers for Flaky AI Dependencies

A breaker sized for a dependency whose calls take seconds and cost money, including which errors should trip it and which must not.

7 min read

Fallback Chains: What to Do When a Model Is Down

How to order a chain of models, what breaks when the second one takes over, and why the quality cliff is the part that hurts rather than the outage.

7 min read

Provider-Agnostic Code: The Interface That Survives a Swap

Where to draw the abstraction boundary so that changing model or vendor is a configuration change, and which leaks are not worth sealing.

7 min read

Streaming Through Your Own Backend Without Breaking It

Why a token stream arrives all at once after you put your own server in the middle, which layer is buffering, and how errors and cancellation work once you have already sent a 200.

7 min read

WebSockets vs SSE vs Polling for AI Responses

A transport choice decided by reconnection and resumability rather than by throughput, because the thing being streamed is expensive and cannot be regenerated identically.

6 min read

Handling Partial Failures in a Multi-Step Workflow

Checkpointing and compensation for a pipeline whose steps cannot be replayed to reproduce their outputs, without adopting a workflow framework.

7 min read

Caching Layers in an AI App: What to Cache Where

Four distinct caches with four different keys and four different ways of going wrong, plus a hit-rate model you run against your own request distribution.

7 min read

Concurrency Control: Not Melting Your Own Rate Limit

Semaphores, token buckets and adaptive concurrency for an API with two independent limits and a service time that varies by an order of magnitude.

7 min read

Backpressure and Queue Depth in AI Pipelines

How to decide what to shed and when, using estimated wait rather than queue depth, before the queue itself becomes the outage.

6 min read

Testing an AI Feature Without Calling the Model

The seams that make the expensive, non-deterministic part swappable, and the surprisingly large amount of your feature that is ordinary testable code.

6 min read

Mocking and Recording LLM Responses in Tests

Cassette-style recording for model calls: what to match on, how to record a stream, what to redact, and the re-record workflow that keeps cassettes from rotting.

7 min read

Local Development Against Expensive APIs

Three development modes, the switch between them, and the spend guardrails that stop a loop left running overnight from becoming a story.

6 min read

Secrets Management for AI Applications

The six places an inference credential leaks that a normal secrets checklist does not cover, and the controls that contain the damage when one does.

6 min read

Normalising Errors Across Heterogeneous APIs

One error taxonomy over several vendor dialects, organised by what the caller should do rather than by what the vendor called it.

6 min read

Designing a Retry Policy That Doesn’t Double-Charge

Which failures are free to retry and which are purchases, how to bound retries by spend rather than by count, and why the timeout is the dangerous case.

7 min read

Progressive Enhancement: AI Features That Work When AI Doesn’t

The unfashionable argument for building the deterministic path first, and what it buys you beyond availability.

6 min read

Other topics

Building reliable AI applications · Multigrid