Handling Spot Interruption for a Fargate-Backed Batch Job

Fargate Spot will take your task away. AWS tells you two minutes beforehand, and the entire design problem is deciding what your batch job has to have finished writing by the time that window closes. Most jobs that lose work on Spot do not lose it because the warning was missed — they lose it because the warning was received by the wrong process, or because the container was killed thirty seconds in.

What the two minutes actually gives you

AWS documents the interruption contract precisely. When Fargate reclaims Spot capacity, a two-minute warning is sent before the task stops, delivered two ways at once: as an ECS Task State Change event to Amazon EventBridge, and as a SIGTERM signal to the running task. The event carries stopCode of SpotInterruption and a stoppedReason of “Your Spot Task was interrupted.” That is from Amazon’s Amazon ECS clusters for Fargate documentation.

Now the part that costs people work. The two minutes is the warning period, not the grace period. How long your container is allowed to keep running after SIGTERM is set by stopTimeout in the container definition, and AWS documents its default as 30 seconds, with a maximum of 120. A task definition that does not set stopTimeout gets thirty seconds and then a SIGKILL, regardless of the fact that AWS was prepared to wait two minutes. If your shutdown path involves flushing a buffer of model responses to S3, thirty seconds is the difference between a clean checkpoint and a corrupt one.

"containerDefinitions": [
  {
    "name": "batch-inference",
    "image": "111122223333.dkr.ecr.us-east-1.amazonaws.com/batch-inference:9f2c1a",
    "essential": true,
    "stopTimeout": 120,
    "command": ["python", "-u", "worker.py"]
  }
]

Receiving SIGTERM in the container

AWS is explicit that the signal must be received from within the container to do anything useful, and this is where a correct stopTimeout still ends in a SIGKILL. Signals go to PID 1. If your Dockerfile uses the shell form — CMD python worker.py — then PID 1 is /bin/sh, the shell does not forward SIGTERM to its child, and your Python process never hears about the interruption. Use the exec form, or set initProcessEnabled in the task definition’s linuxParameters so that a real init process reaps and forwards.

The handler itself should do as little as possible. Set a flag; do not try to finish the current batch inside the signal handler.

import signal, sys, json, boto3

draining = False

def on_sigterm(signum, frame):
    global draining
    draining = True

signal.signal(signal.SIGTERM, on_sigterm)

s3 = boto3.client("s3")
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")

for row in work_items(start_cursor):
    if draining:
        checkpoint(row.index)      # write the cursor, then leave
        sys.exit(0)                # exit 0: this is a planned stop
    resp = bedrock.converse(
        modelId="us.anthropic.claude-sonnet-4-5-20250929-v1:0",
        messages=[{"role": "user", "content": [{"text": row.prompt}]}],
    )
    emit(row.id, resp["output"]["message"]["content"][0]["text"])

Exiting 0 rather than letting the kill land matters for what you see afterwards: a task that exits cleanly on SIGTERM is distinguishable in your logs from one that ran out of time, and only the second kind needs investigating.

Checkpointing a batch inference job

A batch of model calls has an unusual shape for checkpointing: each unit of work is slow (hundreds of milliseconds to tens of seconds), expensive, and independent. That combination argues for checkpointing after every item rather than every N items, because re-running one item after an interruption costs you one model call, while re-running a thousand costs you a thousand.

• Write the result before you advance the cursor. If the cursor moves first and the task dies, that item is silently skipped and nobody notices until the output is short.
• Make the write idempotent. A conditional put keyed on the input row id means a replayed item overwrites itself rather than duplicating. The same reasoning applies to the model call — see idempotency keys on a model request.
• Do not buffer results in memory across many items. A 30-second shutdown budget will not flush a large buffer, and the larger the buffer the more you lose when it does not.
• Keep the shutdown path free of network calls you cannot bound. A final S3 put with a short timeout is fine. Draining an in-flight model call that may take 40 seconds is not, if your stopTimeout is 30.

If the work arrives from a queue rather than a numbered range, the checkpoint is the queue itself: do not delete the message until the result is durable, and size the visibility timeout so an interrupted task’s messages come back promptly. That interaction is the subject of visibility timeout for model calls.

Seeing the interruption from outside

The EventBridge event is the half of the warning your container cannot give you: it tells the rest of your system that capacity was reclaimed, which is what you want for metrics and for deciding whether to fall back to on-demand. AWS notes that Fargate does not automatically replace Spot capacity with on-demand — the service scheduler retries on Spot, and a service with a single task simply stays interrupted until capacity returns.

{
  "source": ["aws.ecs"],
  "detail-type": ["ECS Task State Change"],
  "detail": {
    "clusterArn": ["arn:aws:ecs:us-east-1:111122223333:cluster/batch"],
    "stopCode": ["SpotInterruption"]
  }
}

Point that rule at a metric filter and you get an interruption rate you can reason about. If a job is being interrupted several times an hour in one Availability Zone, the fix is usually a capacity provider strategy with a non-zero base on FARGATE and the remaining weight on FARGATE_SPOT, so a floor of the work always runs on capacity nobody can reclaim. The same trade-off on GPU hardware is covered in spot GPU strategy.

Putting it together

1. Make PID 1 your process: use the exec form of CMD, or set linuxParameters.initProcessEnabled to true.
2. Set stopTimeout to 120 in the container definition. The default of 30 is the single most common reason a correct handler still loses work.
3. Install a SIGTERM handler that sets a flag, and check that flag at the top of the work loop rather than inside a model call.
4. Write each result durably before advancing the cursor, and key the write on the input id so a replay is a no-op.
5. Run the task with a capacity provider strategy naming FARGATE_SPOT, and add an EventBridge rule matching stopCode of SpotInterruption so interruptions become a metric instead of a mystery.
6. Verify the shutdown path deliberately: run the task on-demand and send SIGTERM yourself with aws ecs stop-task --task <arn>, then confirm the checkpoint is where you expect and the exit code is 0.

That last step is the one people skip, and it is the only one that proves the other five. A Spot interruption is not a good moment to discover that your handler was never wired up.