Spot and Preemptible GPUs Without Losing Work

Spot capacity is cheaper because it can be taken back. Whether that trade is worth making is not a matter of opinion: it is a short piece of arithmetic involving the discount, your interruption rate, how long a checkpoint takes to write and how long a restart takes to get back to speed. This page derives it, then gives the drain handler that makes the numbers hold.

What interruption actually looks like

Every major cloud offers discounted capacity that the provider may reclaim, under various names — spot, preemptible, low-priority. The mechanism is consistent even where the branding is not:

1. The provider decides to reclaim the instance, for capacity or price reasons you cannot see.
2. A termination notice is published, usually to the instance metadata service on the machine itself, and often mirrored as a platform event. Your code learns about it by polling the metadata endpoint or subscribing to the event.
3. A grace period elapses. It is short — the point is that it is measured in a small number of minutes, not hours.
4. The instance is stopped or terminated.

On Kubernetes the notice usually reaches you as a node condition or a taint applied by a termination-handler DaemonSet, which cordons the node and evicts pods. Your pod then sees a normal SIGTERM within its terminationGracePeriodSeconds, which means the drain logic from Kubernetes for model serving is the same code path. Keep the grace period comfortably shorter than the provider’s notice window, or the node disappears mid-drain.

The saving, derived

Write down what you are choosing between. All of these are yours to fill in; none of them is a number this page can know.

Labelled assumptions

  P_on    on-demand price of the instance, per hour
  d       spot discount, as a fraction (0.7 means spot costs 30% of on-demand)
  P_sp    spot price = P_on × (1 − d)
  M       mean time between interruptions for this instance type,
          in the region and zone you are actually in, in hours
  C       cost of writing one checkpoint, in wall-clock hours
          (checkpoint size ÷ write throughput; often 30–120 s = 0.008–0.033 h)
  R       restart cost after an interruption, in wall-clock hours:
          acquire replacement + boot + pull image + load weights + reach
          steady state
  W       useful compute hours the job needs if never interrupted

With checkpoint interval T hours, each interruption loses on average half an interval of progress, plus the restart:

Overhead per interruption ≈ T/2  (lost work)  +  R  (restart)

Number of interruptions over a job of wall-clock length H:
  N = H / M

Checkpoint writes over that job:
  K = H / T,  each costing C

Total wall-clock H solves:
  H = W + K·C + N·(T/2 + R)
    = W + (H/T)·C + (H/M)·(T/2 + R)

Rearranged:
  H = W / ( 1 − C/T − (T/2 + R)/M )

Effective cost against running the same job on on-demand:

  cost_spot     = H × P_on × (1 − d)
  cost_ondemand = W × P_on
  effective discount = 1 − (H/W) × (1 − d)

A worked instance, with every input labelled so you can swap them:

Assumptions: d = 0.70,  M = 8 h,  C = 0.02 h (72 s),  R = 0.25 h (15 min),
             W = 100 h of useful compute,  T = 1 h.

  Denominator = 1 − 0.02/1 − (1/2 + 0.25)/8
              = 1 − 0.02 − 0.09375
              = 0.88625
  H = 100 / 0.88625 = 112.8 wall-clock hours

  cost_spot / cost_ondemand = 112.8/100 × 0.30 = 0.338
  effective discount = 66%,  against a headline 70%.

Now make the interruptions four times more frequent — M = 2 h:

  Denominator = 1 − 0.02 − 0.375 = 0.605
  H = 165.3 h,  ratio = 1.653 × 0.30 = 0.496
  effective discount = 50%.  Still worth it.

Now make the restart expensive — R = 1.5 h, a big model on a cold node,
with M = 2 h:

  Denominator = 1 − 0.02 − (0.5 + 1.5)/2 = 1 − 0.02 − 1.0 = −0.02

The denominator has gone negative, which is the arithmetic telling you the
job makes no net progress: it is interrupted, on average, before it has
finished recovering from the previous interruption. No discount rescues
that. Fix R first — cache weights on the node, keep a warm replacement
pool — or do not use spot for this job.

That last case is the one worth internalising. The variable that most often kills spot economics is not the interruption rate, it is R — how long it takes to get productive again — and R is under your control in a way that M is not.

How often to checkpoint

There is an optimum, and it is a classic result from high-performance computing usually attributed to Young (1974) and refined by Daly (2006). Checkpoint too rarely and you lose too much work per interruption; too often and you spend all your time writing. Minimising total overhead gives:

T_opt ≈ sqrt( 2 × C × M )

  C = time to write one checkpoint (hours)
  M = mean time between interruptions (hours)

With C = 0.02 h and M = 8 h:
  T_opt = sqrt(2 × 0.02 × 8) = sqrt(0.32) = 0.566 h ≈ 34 minutes

With C = 0.02 h and M = 2 h:
  T_opt = sqrt(0.08) = 0.283 h ≈ 17 minutes

Note the square root: quadrupling the interruption rate only halves the
checkpoint interval. The optimum is flat near its minimum, so anything
within roughly a factor of two of T_opt costs you very little.

Because the curve is flat, do not over-engineer this. Round to a sensible number of training steps, and prefer to checkpoint on step count rather than wall time so that runs are reproducible.

What a resumable checkpoint contains

A checkpoint that only holds model weights is not resumable — you will restart with a fresh optimiser and a reshuffled data order, and the loss curve will show it. The complete set:

Write it atomically. The failure mode is being interrupted while checkpointing, which leaves a truncated file that looks valid until it is loaded. Write to a temporary name, fsync, then rename — rename is atomic within a filesystem, and for object storage, upload under a temporary key and copy or complete the multipart upload last. Keep the previous two checkpoints; the cost of the storage is trivial against the cost of discovering the newest one is corrupt.

# The atomic pattern, in outline. Framework-independent.
tmp = f"{ckpt_dir}/step-{step}.tmp"
final = f"{ckpt_dir}/step-{step}.pt"

save_everything(tmp)              # weights, optimiser, step, data cursor, rng
os.fsync(open(tmp).fileno())      # durable before we claim it exists
os.replace(tmp, final)            # atomic within a filesystem
write_pointer(ckpt_dir, final)    # a small "latest" file, also written atomically
prune_all_but_newest(ckpt_dir, keep=2)

The drain handler

The handler has one job: turn a termination notice into a checkpoint before the machine goes away. It must be fast, it must be safe to run at any point in the step loop, and it must be idempotent because the notice can arrive more than once.

# Poll the metadata endpoint for a termination notice, set a flag, and let
# the training loop act on it at a safe point. The URL and the response shape
# are provider-specific: read your provider's current docs and put the value
# in configuration.
import threading, time, urllib.request

NOTICE_URL = os.environ["SPOT_NOTICE_URL"]
POLL_SECONDS = 5
preempting = threading.Event()

def watch():
    req = urllib.request.Request(NOTICE_URL, headers=notice_headers())
    while not preempting.is_set():
        try:
            with urllib.request.urlopen(req, timeout=2) as r:
                if r.status == 200:
                    preempting.set()
                    return
        except Exception:
            pass          # 404 or connection error means "no notice yet"
        time.sleep(POLL_SECONDS)

threading.Thread(target=watch, daemon=True).start()

# In the training loop, at a step boundary — never mid-step:
for step, batch in enumerate(loader, start=start_step):
    train_one_step(batch)
    if preempting.is_set():
        save_checkpoint(step)      # the atomic save above
        mark_run_resumable(step)
        sys.exit(0)                # exit cleanly so the scheduler requeues
    if step % checkpoint_every == 0:
        save_checkpoint(step)

Also catch SIGTERM and set the same flag, because on Kubernetes that is how the notice reaches you. One flag, two sources, one exit path. And measure how long your emergency checkpoint takes: if it is longer than the grace period, the handler is decoration. If it is, checkpoint asynchronously to local NVMe during normal operation and have a sidecar copy the file to object storage, so the emergency path only has to flush.

Spot for inference is a different problem

Training tolerates interruption because progress is checkpointable. Inference has no equivalent — a half-generated response cannot be resumed on another machine, because the KV cache lives in device memory and is not portable. So spot for serving is not a checkpointing problem, it is a capacity problem:

• Run a mixed pool. Enough on-demand or reserved capacity to carry your committed floor of traffic, with spot on top for the peak. If spot vanishes, you degrade to slower rather than failing.
• Drain, do not die. On notice, fail readiness immediately so the load balancer stops sending new requests, then let in-flight generations finish inside the grace period.
• Cap generation length on spot replicas. A pool whose longest possible response is 30 seconds can drain inside almost any notice window; one that allows five-minute agent runs cannot.
• Spread across instance types and zones. Interruptions correlate within a type and a zone, so a pool of one type in one zone can go entirely at once.

When spot is the wrong answer

Three cases, stated plainly. First, when R is large relative to M, as the negative denominator above showed — very large models on nodes that must download weights are the usual instance. Second, when the job has a hard deadline: spot turns a predictable finish time into a distribution, and if you must have the result by Friday you are buying certainty, not compute. Third, when the capacity is scarce enough that replacements are not available — during a shortage the reclaim rate and the replacement wait rise together, which is exactly the correlation the arithmetic above assumes away. The GPU shortage and rent versus buy cover the market side of that.

Track your own M. Log every interruption with its instance type, zone and timestamp; after a month you have a real distribution rather than a vendor’s advertised frequency, and you can put a number in the formula above that is actually about you. That log is also the input to the quarterly capacity review.