Prompt Chaining vs One Big Prompt

Decomposition is usually argued for on taste — smaller prompts are tidier. The decision is actually two calculations, one about tokens and one about compounding error, and they can point in opposite directions on the same task.

What decomposition actually changes

Splitting one call into three changes four things, and only the first is usually discussed.

• Attention budget. Each step attends to a smaller context, so the instruction for that step competes with less.
• Observability. Every intermediate becomes a value you can log, validate and assert on. A monolith gives you one string and a shrug.
• Model choice per step. Extraction and classification rarely need the model that drafts the final answer. This is where most of the cost win comes from, not from the decomposition itself.
• Round trips. Steps are sequential, so latency is the sum of the steps, each with its own time to first token and network overhead.

There is a fifth change, and it shows up in the on-call rotation rather than in the design document: the number of things that can fail independently. Three calls mean three timeouts, three rate limits, three retries and three chances for a provider hiccup, and if the steps are not idempotent, a retry halfway through repeats work you have already paid for. Decomposition is a distributed-systems decision as much as a prompting one.

A worked cost model

Take a concrete task: read a 6,000-token support thread, decide whether it is a refund case, and draft a reply. Assumptions stated up front, because they are the whole calculation — a capable model at $3 per million input and $15 per million output, a small model at $0.50 and $1.50, and no caching. Substitute your own figures from your provider’s pricing page; the arithmetic is what transfers, not the rates.

MONOLITH — one capable-model call
  in  6000 x $3/M   = $0.0180
  out  800 x $15/M  = $0.0120
                      -------
                      $0.0300

CHAIN — three steps, mixed models
  1 extract facts   capable  in 6000  out 300   $0.0180 + $0.0045 = $0.0225
  2 classify        small    in  500  out  50   $0.00025 + $0.000075
  3 draft reply     small    in  350  out 700   $0.000175 + $0.00105
                                                -------
                                                $0.0244

CHAIN — same three steps, all on the capable model
  1 $0.0225   2 $0.0022   3 $0.0116                $0.0363

Two readings. Chaining on one model is more expensive here — about 21% more — because step 1’s big input is paid once and the intermediates are paid again as input downstream. Chaining across models is cheaper than the monolith, by about 19%, and the saving is entirely attributable to running two of the three steps on a cheaper model.

So “chaining saves money” is false as stated. Chaining enables a saving by making cheap steps separable. If every step still needs the frontier model, decomposition costs you tokens and you should be buying something else with them — usually reliability.

One more term belongs in the model: if the same 6,000-token thread is re-sent to several steps, it is a shared prefix, and prompt caching makes the repeat sends cost a fraction of the first. A chain designed so every step shares a prefix behaves very differently from one that re-sends different slices.

The error arithmetic

Independence is the ugly part. If three steps each succeed 95% of the time and each depends on the last, the chain succeeds 0.95³ = 0.857 of the time. A single call that is right 88% of the time beats it. Five 95% steps give 0.774. This is why long agentic chains built from individually excellent steps feel unreliable: the failure rate compounds and nothing about the individual prompts looks wrong.

Chains beat monoliths on accuracy when the steps are not naked. Add a validator that catches a fraction d of a step’s failures and a retry that succeeds with probability r, and the effective per-step success rate becomes p + (1 − p)·d·r. With p = 0.95, a validator catching 80% and a retry succeeding half the time: 0.95 + 0.05·0.8·0.5 = 0.97, and three such steps give 0.913 rather than 0.857.

Independence is also the assumption most likely to be wrong, and it is wrong in one convenient direction and one inconvenient one. Conveniently, a hard case tends to be hard for every step, so failures cluster and the overall rate is better than the product suggests. Inconveniently, a step handed a subtly wrong intermediate is no longer 95% accurate at anything you care about — it is accurate about the wrong input, which is how a chain produces a confident, well-formatted answer to a question nobody asked. Validate the intermediate values, not merely their shape.

That is the real argument for decomposition, and it is conditional: the win comes from the checks, not from the splitting. A chain whose intermediates are unvalidated free text has taken the compounding penalty and bought nothing with it.

When to chain

Building one that stays debuggable

• Give every step a machine-checkable output type. If you cannot write the validator, that boundary is in the wrong place.
• Log the intermediates with a shared request id. Without it, a bad final answer is unattributable and you will start rewriting the wrong prompt.
• Keep an eval set per step, not only end to end. Step-level evals are what let you swap a cheaper model into step 2 with evidence.
• Set a hop budget and a total token ceiling. Chains that can retry are chains that can loop.

One middle ground is worth knowing about, because it wins more often than either extreme: a single call that returns several named fields — a plan, an intermediate extraction, then the final answer — in one structured response. You keep one round trip and one context, and you still get intermediates you can log and assert on. It buys most of the observability and none of the extra token cost, and it fails precisely where a real chain earns its keep: you cannot route the cheap parts to a cheaper model, and you cannot retry one step in isolation.