The Demo-to-Product Gap

A demo proves the model can do the thing once. A product promises it will do the thing every time, on inputs nobody has seen, at a price somebody has to forecast. Almost everything people call “the last ten per cent” is the distance between those two sentences, and it is not ten per cent.

What a demo is allowed to assume

A demo is not a bad artefact. It is a legitimate experiment with an explicit scope, and the reason it is fast is that it is permitted four assumptions that a product is not. Writing them down is more useful than a list of things that broke, because every production surprise you are about to have is one of these four assumptions expiring.

• The inputs are the ones you chose. A demo is driven by examples selected, often unconsciously, because they worked. The product is driven by whatever a user pastes in.
• One sample is the behaviour. The demo ran the prompt a handful of times and the outputs were good. Nobody looked at the tail, because a tail needs hundreds of samples to see.
• Cost is a per-call number. Somebody multiplied a price by a token count. Nobody multiplied a distribution of token counts by a distribution of retries.
• A human is present. The person running the demo reads every output, silently discards a bad one, and rephrases. That person is the quality layer, and they are not shipping with it.

The fourth is the load-bearing one and the least visible. A demo is a human-in-the-loop system in which the human is invisible because they are also the narrator.

The input distribution goes first

The first thing that breaks is almost never model quality. It is that real inputs are longer, messier, multilingual, empty, adversarial, duplicated, or a screenshot of the thing you expected as text. The demo prompt was written against a shape of input that turns out to be a narrow slice of what arrives.

The mechanism is worth stating precisely, because it suggests the fix. A prompt is a program whose behaviour on out-of-distribution input is undefined and unannounced — there is no type error, no exception, no empty result. There is a fluent answer computed from something the author of the prompt never considered. So the failure arrives as content, not as an error, and nothing in your monitoring is watching content.

The cheapest counter is not a better prompt. It is a corpus. Pull fifty to a hundred real inputs — from a beta, from the manual process the feature is replacing, from the support queue — before the prompt is finished, and read them. Most of the specification you are missing is in that pile, and it costs an afternoon. This is the same corpus that becomes your golden dataset, so the afternoon is not spent twice.

Then the variance you never sampled

The second failure is that a model is a distribution and a demo is a draw from it. Five good outputs are consistent with a ninety per cent success rate, which is a fine demo and a bad product if the feature runs a thousand times a day: that is a hundred bad outputs daily, each one landing in front of a specific person who did not consent to being in an experiment.

This is not a subtle statistical point and it is routinely skipped because the demo felt convincing. The correction is mechanical: decide what failure rate is acceptable for this feature, then sample enough to distinguish that rate from a worse one. If you need to tell 95 per cent apart from 85 per cent you need tens of examples; if you need to tell 99 from 97 you need hundreds, and the arithmetic for how many is the subject of the statistics page in the evaluation cluster.

The related trap is that the demo was tuned by the same person who judged it, on the same examples, in the same sitting. That is not evaluation; it is fitting. Keeping a held-out slice you do not look at while iterating is the smallest discipline that turns a demo into evidence.

Then cost, which is a distribution too

The third assumption expires quietly. Somebody computed cost per call from the demo prompt and multiplied by expected volume. Every term in that product is understated, and here is the shape of the arithmetic you should be doing instead. Take your own numbers for each input; the point is the structure, not the values.

cost_per_served_request =
      (input_tokens_p50  * input_price)
    + (output_tokens_p50 * output_price)
    + retry_factor        // 1 + P(retry) * attempts, from your error rate
    + fallback_factor     // share of traffic served by a second, dearer model
    + hidden_calls        // classifier, judge, embedding, guardrail per request
    + rework              // regenerations a user asks for, which are traffic too

worst_case_request = same expression at p99 token counts,
                     which is the number that decides your rate limits
                     and your per-user cap, not the p50.

Three of those terms do not exist in a demo at all. There is no retry policy, no fallback, and usually no second call — the guardrail, the classifier and the judge are all added later, and each one is a call per request that nobody added to the forecast. A feature whose demo cost was one model call can easily reach production as three, and the forecast was never revised because the original number was never written down as a formula that could be updated.

The other half of the gap is that the demo has no p99. Token counts have a long right tail — a user pastes a whole contract — and your capacity, your timeouts and your per-user spend cap are all sized by the tail rather than the median.

A build order that closes the gap

The gap closes cheapest in one particular sequence, because each step makes the next one measurable rather than speculative.

Notice that the model does not appear until step five, and the prompt barely appears at all. That is the actual content of the demo-to- product gap: the demo was about the model, and the product is mostly about everything that surrounds it. The engineering half of that surround — deadlines, retries, degradation — is a separate body of work and is well understood; the part above is the part teams skip because it does not feel like building.