Spinning Up a Spot GPU Instance as an Ephemeral CI Runner

An ephemeral spot runner is the answer to the arithmetic on the previous page: below a few dozen GPU jobs a day, you want to pay for the job’s minutes and nothing else. The mechanism is a single-use runner credential, an instance that consumes it, and a termination that is the instance’s own responsibility.

The shape of the thing

Three actors. A cheap orchestrating job on a standard runner mints a runner credential and launches an instance. The instance boots, starts the runner agent with that credential, and picks up exactly one job. When the job ends, the runner deregisters itself and the instance terminates.

The property that makes this safe is that every step is single-use. A long-lived self-hosted runner accumulates state between jobs — Docker layers, checked-out repositories, environment variables, and anything a previous job left behind — which is a correctness problem before it is a security one. An instance that dies after one job cannot leak anything into the next.

The property that makes it cheap is that spot capacity is dramatically discounted against on-demand, and the workload tolerates interruption in a way most workloads do not: the worst outcome of losing a CI job mid-run is running it again.

Just-in-time registration

The older pattern uses a registration token from POST /repos/{owner}/{repo}/actions/runners/registration-token, passed to config.sh --token. It works, but the token is valid for an hour and can register any number of runners, so it is a credential you would rather not put in instance user data.

The just-in-time endpoint is the better shape. You describe the runner you want and GitHub returns a pre-baked configuration bound to that one runner:

curl -sS -X POST \
  -H "Authorization: Bearer $GH_TOKEN" \
  -H "Accept: application/vnd.github+json" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  https://api.github.com/repos/OWNER/REPO/actions/runners/generate-jitconfig \
  -d '{
        "name": "gpu-ephemeral-'"$GITHUB_RUN_ID"'",
        "runner_group_id": 1,
        "labels": ["self-hosted", "linux", "gpu", "spot"]
      }'

The response carries encoded_jit_config, an opaque base64 string. The instance passes it straight to the runner agent — there is no config.sh step at all, because the configuration is already inside the blob:

./run.sh --jitconfig "$ENCODED_JIT_CONFIG"

A runner configured this way is ephemeral by construction: GitHub deregisters it automatically once it has processed one job. There is an org-level equivalent at POST /orgs/{org}/actions/runners/generate-jitconfig. The calling credential needs administration rights on the repository or the self-hosted-runners permission on the organisation, which is another reason to mint it in the orchestrating job rather than store it on the instance.

The instance and its user data

1. Build an AMI with the NVIDIA driver, the container toolkit and the runner agent already unpacked in /opt/actions-runner. Installing a GPU driver at boot adds minutes to every job and is the single biggest avoidable cost in this design.
2. Create a launch template pinning the AMI, the instance type, an instance profile, and a security group with no inbound rules — the runner dials out, nothing dials in.
3. Launch with a spot market request. Setting no maximum price defaults to the on-demand price as the ceiling, which is the right default: you are buying availability, not chasing the floor.
4. Pass the JIT config through user data, base64-encoded by the CLI.

aws ec2 run-instances \
  --launch-template LaunchTemplateName=gpu-ci-runner,Version='$Latest' \
  --instance-type g5.xlarge \
  --instance-market-options 'MarketType=spot' \
  --instance-initiated-shutdown-behavior terminate \
  --tag-specifications 'ResourceType=instance,Tags=[
      {Key=Name,Value=gpu-ci-runner},
      {Key=team,Value=ml-platform},
      {Key=ci-run-id,Value='"$GITHUB_RUN_ID"'}]' \
  --user-data "$(printf '#!/bin/bash\nset -euo pipefail\ncd /opt/actions-runner\nsudo -u runner ./run.sh --jitconfig %s\nshutdown -h now\n' "$JIT")"

--instance-initiated-shutdown-behavior terminate is the quiet hero of that block. With it set, the shutdown -h now at the end of user data terminates the instance rather than stopping it — and a stopped GPU instance still bills for its EBS volume while contributing nothing. The tags are not decoration either; they are what makes this spend show up in the split described in the cost allocation tagging page.

Terminating itself

User data running to completion covers the happy path. It does not cover the job that never arrives, the runner agent that crashes before reaching the shutdown line, or the orchestrating workflow that is cancelled after launching the instance. Every one of those leaves a GPU instance running indefinitely, which is exactly the bill this design was supposed to avoid.

Two backstops, and you want both. First, a hard deadline inside the instance, set before the runner starts:

# in user data, before ./run.sh
shutdown -h +45 "ephemeral CI runner deadline"

Second, a scheduled sweep that terminates any instance tagged Name=gpu-ci-runner older than an hour. The instance can also terminate itself explicitly if you prefer that to shutdown, by reading its own id from the instance metadata service — note that IMDSv2 requires fetching a token first:

TOKEN=$(curl -sX PUT http://169.254.169.254/latest/api/token \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
IID=$(curl -sH "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/instance-id)
aws ec2 terminate-instances --instance-ids "$IID"

That path needs ec2:TerminateInstances on the instance profile. Scope it with a condition on the instance tag so the runner can terminate itself and nothing else.

What a spot interruption does

Spot capacity is reclaimable. AWS publishes an interruption notice through the instance metadata service at /latest/meta-data/spot/instance-action, which returns 404 until an interruption is scheduled and then returns a JSON document with the action and a timestamp. The documented warning window is two minutes. There is also a rebalance recommendation signal at /latest/meta-data/events/recommendations/rebalance, which arrives earlier and less reliably.

Two minutes is not enough to finish a GPU eval, so do not try to drain gracefully. The useful behaviour is to make the failure legible: poll the endpoint from a background loop, and when it fires, write a clear marker to the job log before the instance disappears. Otherwise the job simply stops mid-step and the run page shows a lost-communication error that looks like a bug in your code.

• Retry at the workflow level, not the step level. An interrupted runner is gone; the retry has to provision a new one.
• Do not put releases on spot. Reserve interruption tolerance for jobs whose only cost of failure is time.
• Diversify instance types if you can. A launch template that accepts several GPU instance types has materially better capacity odds than one pinned to a single type in a single availability zone.
• Watch for repeated instant interruptions. An instance reclaimed within seconds of launch, repeatedly, is a capacity signal for that type and zone — not something a longer timeout will fix.