Handling Partial Failures in a Multi-Step Workflow

Five steps, four of them expensive, and the fourth one failed. The classical answer is to retry from the start or to compensate backwards. Both assume you can reproduce what the first three steps did, and with a model in the pipeline you cannot.

Step three failed. Now what?

Take a document pipeline: extract text, chunk and embed, generate a summary, extract structured fields, write the record. Step four fails with a timeout. The options in the textbook are:

• Retry the whole workflow. Correct if every step is idempotent and cheap. Here steps two and three are neither, so this option pays twice for work that already succeeded.
• Compensate backwards. Undo the effects of steps one to three, report failure, let the caller decide. Correct when the steps have external effects that must not linger. Expensive when they do not, because you are throwing away good work.
• Resume forwards from step four. The one you actually want. It requires that the outputs of steps one to three survived the failure of step four, which is the entire design question.

Why you cannot just re-run it

Deterministic pipelines have a cheap escape: instead of storing intermediate results, re-execute the earlier steps to reconstruct them. Pure functions of durable inputs make this free, and it is why much event-sourcing and workflow advice treats replay as the default recovery mechanism.

A model call breaks the assumption twice over. It is not a function of its inputs — the same prompt yields a different summary — and even if you pin temperature to zero, a provider may change the served model, change quantisation, or route you to different hardware, none of which you are told about. So replaying step three does not restore the state the pipeline was in; it produces a different state that looks similar.

That matters when step four consumed step three’s output. If step four extracted fields from a summary, and you regenerate the summary before retrying step four, the fields now describe a summary that no longer exists anywhere. Any artefact already shown to a user, already stored, or already used to make a decision is now inconsistent, and nothing in the system will tell you.

The rule that follows is short: the output of a non-deterministic step is durable state, not a recomputable value. Write it down before you use it.

Checkpoint the outputs, not the progress

A progress marker — “we got to step 3” — is not a checkpoint. On resume you know where you were and not what you had. A checkpoint stores the actual output of each step, keyed by the run and the step, written in the same transaction as the state change that marks the step complete.

create table run_step (
  run_id      uuid not null,
  step        text not null,        -- stable name; renaming a step orphans its checkpoints
  attempt     int  not null,
  state       text not null,        -- succeeded | failed | compensated
  output      jsonb,                -- the actual result, not a pointer to a rerun
  cost_cents  numeric not null default 0,
  route       text,                 -- which model produced it
  created_at  timestamptz not null default now(),
  primary key (run_id, step)
);

Two fields there are easy to skip and expensive to add later. route records which model produced the output, so that after a provider incident you can find every artefact generated by the degraded rung and decide whether to regenerate. And cost_cents per step is what lets you answer whether the retry policy is costing more than the feature earns.

A step runner in forty lines

type Step<T> = {
  name: string;
  run: (ctx: RunCtx) => Promise<T>;
  compensate?: (output: T, ctx: RunCtx) => Promise<void>;
};

async function runWorkflow(runId: string, steps: Step<unknown>[], ctx: RunCtx) {
  const done: { step: Step<unknown>; output: unknown }[] = [];

  for (const step of steps) {
    const saved = await loadCheckpoint(runId, step.name);
    if (saved?.state === "succeeded") {          // resume forwards
      done.push({ step, output: saved.output });
      continue;
    }
    if (ctx.deadline.expired()) throw new Abandoned(step.name);

    try {
      const output = await step.run(ctx);
      await saveCheckpoint(runId, step.name, output, ctx.lastRunMeta);
      done.push({ step, output });
    } catch (error) {
      if (!isFatal(error)) throw new Retryable(step.name, error);   // resume later
      for (const d of done.reverse()) {          // unwind only on fatal
        await d.step.compensate?.(d.output, ctx);
        await markCompensated(runId, d.step.name);
      }
      throw error;
    }
  }
  return done;
}

The distinction between Retryable and fatal is the design. Retryable means the run is parked and resumed later from the same step, keeping every checkpoint — this is the common case and it costs nothing but a delay. Fatal means the run cannot proceed at all, and only then do you unwind. Compensating on a transient timeout is how teams end up undoing three expensive steps because a provider had a bad five seconds.

Compensation, and where side effects go

Compensation is easy for effects you own — delete the row, release the reservation, mark the draft discarded. It is impossible for effects you have released into the world: an email sent, a webhook delivered, a message posted, a payment captured. There is no unsend.

The discipline is ordering. Put every irreversible external effect at the end of the workflow, after everything that can fail. If two external effects are unavoidable, the one that is hardest to reverse goes last. This is not novel advice, but it is worth restating in this context because model-driven pipelines invite the opposite pattern: generate a message, send it, then continue processing. Once the message is gone, the rest of the workflow has no safe failure.

Money spent on generation is itself an unreversible effect, and it is the one people forget to model. If step two spent real money and step four is going to fail permanently, compensation cannot recover that; it can only record it. Keeping the per-step cost on the run is what makes “our failed runs cost more than our successful ones” a discoverable fact rather than a suspicion.

When to reach for a real engine

The runner above is deliberately small, and for a linear pipeline of a handful of steps it is enough. Durable execution engines exist and they are good; the question is whether you need one. Signals that you do:

• Steps that wait for external events — a human approval, a callback days later — where holding the run in memory is not an option.
• Fan-out and fan-in with partial success semantics, where you need “continue with the eighty of a hundred that worked” as a first-class outcome.
• Workflows that must survive a deployment mid-run, which requires versioning the workflow definition itself rather than just the checkpoints.

Below that threshold, a table, a loop and an honest distinction between retryable and fatal will carry you a long way, and they leave the failure semantics visible in code you can read.