The Pilot-to-Production Gap

The gap between a working pilot and a working product is usually described as maturity, which explains nothing. It is four specific changes, each of them a multiplication, and each of them predictable before you start.

Four things change, and only four

A pilot and a production feature run the same code against the same model. What differs is the environment around them, in four ways.

• Volume. More requests per day, by a factor you can write down. Everything proportional to volume — spend, rate-limit pressure, the absolute number of bad outputs — moves by that same factor.
• Variety. A pilot cohort is homogeneous by construction. Production contains the other departments, the other languages, the customer who pastes a spreadsheet into a text box, and the account with eleven years of history.
• Audience. Pilot users chose to be there and will forgive a bad answer. Production users did not choose and will not. The same output quality produces different outcomes.
• Permanence. A pilot is allowed to be down. A feature in the product has an implicit availability promise the moment it appears in the interface, and that promise is what turns a model outage into an incident.

Everything below is a consequence of one of those four. If a readiness discussion is not tracing back to one of them, it is probably not about readiness.

Notice what is not on the list: the model, the prompt and the architecture. Those are usually unchanged between the pilot and the rollout, which is exactly why the gap is confusing from the inside — the thing you built still works, and it is being asked a different question. Framing it this way also makes the work estimable, because each of the four changes maps to a specific set of build items rather than to a vague instruction to harden the system.

Rare becomes daily

This is the change that surprises people who did everything else right, and it is pure arithmetic. Suppose the pilot ran 400 requests a week and something bad happens on 1 in 500 requests. That is slightly less than one occurrence per week — plausibly zero across the whole pilot, and therefore invisible.

rate of a "rare" failure:  p = 1/500 = 0.002

pilot     400 requests/week   ->  0.8 events per week
                                  P(zero events in a 4-week pilot)
                                  = (1 - 0.002)^1600 = 4.1%

production  40,000 requests/week -> 80 events per week
                                    ~11 per day

Note the second line: a four-week pilot at that volume had only about a 4% chance of showing zero events, so this particular failure probably was seen and dismissed as a one-off. The dangerous ones are rarer still. A 1-in-20,000 failure has a 92% chance of never appearing in that pilot and appears roughly twice a week in production, and it is exactly the class of event — a leaked instruction, a wrong customer name, an answer that invents a policy — that produces a meeting.

The consequence for design is that production needs mechanisms the pilot did not: sampling and grading of live traffic rather than eyeballing, alerting on rates rather than on incidents, and a way to find the specific request afterwards. That is the difference between grading production traffic and running an eval set once, and it is a build item, not a disposition.

The tail becomes the experience

In a demo you watch a handful of requests and they feel fast. At production volume the number that decides how the feature is perceived is not the median; it is the slow end, because every user hits it occasionally and the ones who hit it are the ones who complain.

Three tails matter and they are separate numbers. Latency: measure p95 rather than a mean, because a single cold start drags a mean past every request anyone actually experienced. Cost: the expensive requests are not the average ones, and a per-request ceiling has to be set from the p99 rather than from the mean or it will start rejecting real traffic. Quality: model failures cluster on the hard inputs, which usually means they cluster on one customer, one document type or one language — an aggregate failure rate can look fine while one segment is unusable.

Availability joins the list here too, because permanence changed. A model provider will have a bad hour at some point during the year, and production needs a decided answer for what the interface does then — which is the difference between a feature that degrades and one that shows a spinner until the user leaves.

The data stops being the demo data

Most pilots run on a curated slice: an export somebody cleaned, one team’s documents, one customer’s records, everything in one language, no access control because the pilot team could see everything anyway. Production has none of those properties, and the differences are not cosmetic.

• Permissions become part of retrieval. The moment two users must see different documents, the index needs the access model in it, and getting that wrong is a data leak rather than a quality problem — the patterns are the ones in multi-tenant retrieval.
• Freshness becomes a pipeline. A one-off index is fine for six weeks and wrong by month three. Somebody has to own updates, and “we will reindex nightly” is a job with failure modes.
• Personal data arrives whether or not you planned for it. Free-text fields will contain names, account numbers and occasionally something special-category, which turns what you log into a compliance question rather than a debugging convenience.
• The long tail of format arrives too. Scanned PDFs, forwarded email chains, tables pasted as text, a document in the wrong language. The pilot corpus was tidy because a person chose it.

The readiness gate

A short gate, run before the rollout rather than at the end of it. Each row asks for evidence, not for confidence, and the evidence is a link or a number.

The rollback row is the one worth insisting on hardest. Every other row reduces the chance of a problem; that one bounds the cost of the problems you did not anticipate, and it is the reason a staged rollout is worth the extra fortnight. A feature you can turn off is allowed to be imperfect. A feature you cannot turn off has to be right, and nothing involving a language model is going to be right on the first day at full volume.