Eight Things That Surprise Backend Engineers About AI Systems

Almost everything an experienced backend engineer knows transfers directly to an LLM system: the same idempotency problems, the same retries, the same tenancy questions, the same ledgers. Three models do not transfer — the cost model, the failure model and the testing model — and the eight surprises below are all consequences of those three.

The skills transfer; three models do not

This is worth saying first because the usual framing — that AI engineering is a new discipline — is mostly wrong and leads people to discard experience that is directly applicable. A gateway that bills per request, holds balances and reconciles a ledger is an ordinary transactional system with an unusual dependency.

The unusual dependency is the whole of the difference: it charges per call, it can fail while returning success, and it is not stable over time.

1. Cost is per request, and unbounded by default

In ordinary backend work, a bug produces load. Here it produces an invoice. A retry loop against a normal service costs CPU; a retry loop against a model costs money at a rate set by somebody else, and there is no natural back-pressure because the provider is happy to serve.

The consequences that catch people out:

• A request must reserve funds before it runs, not reconcile afterwards, or a burst can overdraw an account.
• The reservation must be an upper bound over every route the request could take. Pricing the hold on the cheapest candidate means any failover overdraws by the difference.
• Two different token-limit parameters may both be forwarded upstream. If one is used for the hold and the other for the charge, the two disagree by design.
• An aborted stream still cost what the provider generated. Billing that reads usage only from a terminal message charges nothing for a cancellation the provider bills in full.

The general shape is denial of wallet: a class of vulnerability with no equivalent in ordinary services, where the attack is simply to use the system.

2. Correct is a distribution, not a value

The model returns a probability distribution over next tokens and something outside it samples one. So the same input can produce different output, and “it worked when I tried it” carries almost no information.

Two practical consequences. First, any assertion about output must be a property — it parses, it validates, it contains the required field — rather than an equality. Second, bug reports need the stored request and response, because the reporter cannot reproduce it and neither can you. Even at temperature zero, identical output is not guaranteed across infrastructure changes; why temperature zero is not deterministic explains why.

3. HTTP status does not classify the failure

In ordinary integration work, a 400 means the caller sent something wrong and a 429 means slow down. With model providers, both can mean “the account behind this integration has no credit”, and providers do not agree on which.

Reading it as 400 returns a billing problem to the end user as though their input were malformed, and skips the healthy alternative routes. Reading it as 429 retries with backoff against an account that will refuse every attempt. Classification has to read the response body, and the matching has to be narrow enough that a genuine bad request is never converted into a failover. The full classifier is in production LLM error classes.

4. The worst failures return 200

Malformed output and wrong output are both successful HTTP responses. A dashboard built on status codes reports a health it cannot see.

There is a third silent case that surprises people even after they have absorbed the first two: failure hidden by successful failover. When a provider refuses and the request is served by the next route, the user gets a 200 and the log row names whichever provider answered. The failure is invisible because the failover worked, and the first symptom is a larger bill, because the fallback is usually the dearer route. Record the refusal at the moment it is classified, not at the end of the request.

5. The dependency changes underneath you

A library version is fixed until you change it. A model identifier is not necessarily fixed at all: an alias can be repointed, a quantisation can be swapped, a deprecated model can be redirected to its successor. The same request can produce systematically different output with nothing in your repository having changed.

The defence is to pin explicit versions where the provider offers them, record the model string the response reports and alert on a change where it does not, and keep a small regression set that runs on a schedule against the live endpoint. See silent model updates.

6. The input is the program

There is no state between calls. Everything the model can use is in the request, every time, which has two consequences that feel wrong to somebody used to sessions and databases.

• The prompt is code and needs versioning, review and a changelog. A prompt edited in a dashboard by somebody outside the repository is a production deploy with no history.
• Assembling the request is where most bugs live. The template is rarely the problem; the retrieved documents, the truncated history, the tool schemas and the ordering are. Debugging means logging the assembled request, not the template — and most teams cannot do that on demand.

7. Untrusted text reaches the control path

This is the one with no analogue in ordinary backend work. In a normal system, data and instructions live in different places: a query is parameterised, so the data cannot become the query. A model has one channel, so any text that reaches the context can attempt to instruct the model.

Every retrieved document, every web page fetched by a tool, every uploaded file and every user message is in that channel. Indirect prompt injection is the exploit; the mitigation is architectural rather than textual, because there is no reliable equivalent of parameterisation. Constrain what the model is able to do — capabilities, allow-lists, confirmation steps — rather than instructing it not to.

The same property produces a second surprise: a rule authored by a customer and executed by you is a customer-supplied program running on your infrastructure. A regular expression is the common case, and a pattern that backtracks catastrophically does not slow down the tenant who wrote it — it stalls the process serving everyone.

8. Tests assert properties, not values

The whole testing model shifts, and four habits carry over badly.

The complement to the suite is an evaluation set: building an eval harness covers the mechanics, and what a passing suite cannot see covers what neither of them catches.