Fargate Task Startup Time for On-Demand Inference

A Fargate task is a small virtual machine with its own network interface, created for this task and thrown away afterwards. That is why it starts more slowly than a container scheduled onto a warm EC2 instance, and the gap is not one delay but a sequence of them, each visible as a lifecycle state.

The lifecycle states

Amazon documents the task lifecycle as a sequence of states: PROVISIONING, PENDING, ACTIVATING, RUNNING, then DEACTIVATING, STOPPING and STOPPED on the way down. The ECS task lifecycle documentation describes each. Three of them contain the startup cost.

• PROVISIONING — ECS performs steps before the task is launched. The documentation gives the example that matters here: for tasks using the awsvpc network mode, an elastic network interface must be provisioned. Every Fargate task uses awsvpc.
• PENDING — a transition state where ECS waits on the container agent, and where the task waits if resources are not yet available.
• ACTIVATING — after launch and before RUNNING. This is where ECS pulls the container images, creates the containers, configures task networking, registers load balancer target groups and configures service discovery.

Reading that list is most of the explanation. On a warm ECS instance with EC2 launch type, the image is often already cached on the host and no network interface has to be created. Fargate gives you a fresh, isolated environment on every task, and the price of the isolation is that none of that work is amortised.

PROVISIONING: the network interface

Creating and attaching an ENI is an EC2 control plane operation, and it is the part of a Fargate launch you have the least influence over. What you do influence is whether it succeeds quickly, and there are a few ways to make it slow or fail outright.

• Subnet address exhaustion. One ENI per task, one IP per ENI. A /27 subnet running a service that scales to fifty tasks runs out, and the failure surfaces as tasks stuck before they start rather than as a clear capacity error.
• No route to the registry. A task in a private subnet needs a NAT gateway or VPC endpoints for ECR API, ECR DKR, S3 and CloudWatch Logs. Without them the task provisions successfully and then hangs in ACTIVATING failing to pull, which reads as a slow start until it times out.
• Interface endpoints are worth having anyway. Pulling through a VPC endpoint keeps image traffic off the NAT gateway, which is both a latency and a per-gigabyte cost consideration for large inference images.

ACTIVATING: pulls, wiring and registration

The image pull is the part that scales with your decisions, and for an inference image it is usually the largest single component. Nothing is cached between tasks on a fresh Fargate environment, so every task transfers and decompresses the layers it needs.

Two smaller items in the same state deserve attention because they are invisible in the container logs. Registering with a load balancer target group is not instantaneous: the target has to pass the target group’s health check threshold before traffic arrives, so a HealthCheckIntervalSeconds of 30 with a HealthyThresholdCount of 3 adds up to a minute and a half after the container is already serving. For a model container that takes a while to load weights, this is often the difference between a launch that feels slow and one that feels broken, and it is a target group setting rather than anything about Fargate. Service discovery registration adds DNS propagation on top when it is in use.

Making the image part faster

AWS has shipped two mechanisms for this, both of which apply without changing the application.

Seekable OCI. SOCI is an image format extension that lets Fargate start a container without downloading the whole image first, by building an index over the files in an existing image so individual files can be extracted without the full download. AWS’s announcement states that when starting a task, Fargate automatically detects whether a SOCI index exists for the image and starts it without waiting for the full download, at no additional cost beyond storing the index in ECR. The index is built separately from the image and pushed alongside it, so this is a CI pipeline change rather than a Dockerfile one.

zstd compression. Fargate platform version 1.4.0 and later use containerd, which supports zstd-compressed layers. AWS reported up to a 27% reduction in task and pod startup times from zstd compression, with the largest images benefiting most, in a containers blog post from October 2022. It is a buildx output option rather than a Dockerfile change:

docker buildx build \
  --output type=image,name=ACCOUNT.dkr.ecr.eu-west-1.amazonaws.com/inference:1.4.0,compression=zstd,compression-level=3,force-compression=true,push=true .

That post is the source for both the figure and the flags. Note force-compression=true: without it, layers that arrive already gzip-compressed from the base image are passed through untouched, and most of the image stays gzip.

The third lever is the ordinary one, and it is not about compression at all: do not put the model weights in the image if they change on a different cadence from the code. A 500 MB service image that fetches 8 GB of weights from S3 at startup has moved the transfer rather than removed it, but it has made the code deploy cheap and it lets you use S3 transfer parallelism, which a layer pull does not give you.

Finding out where the time went

ECS records timestamps on the task itself, and this is the part most discussions of Fargate startup skip. DescribeTasks returns createdAt, connectivityAt, pullStartedAt, pullStoppedAt and startedAt. The differences between them decompose a launch into exactly the stages above.

aws ecs describe-tasks \
  --cluster inference \
  --tasks arn:aws:ecs:eu-west-1:111122223333:task/inference/abc123 \
  --query 'tasks[0].{created:createdAt,connectivity:connectivityAt,pullStart:pullStartedAt,pullStop:pullStoppedAt,started:startedAt}'

createdAt to connectivityAt covers provisioning and networking. pullStartedAt to pullStoppedAt is the image pull, in isolation, which settles the argument about whether the image is the problem. pullStoppedAt to startedAt is your application initializing — loading weights, warming a tokenizer, building a client. Sample a handful of launches and the distribution across those three intervals tells you which of the sections above to read again.

The last thing to consider is whether the shape fits at all. Fargate starts a fresh isolated environment for every task, and no amount of image work makes that free. If the workload is bursty inference where the first request cannot wait, the answer is usually a minimum task count on the service — the same trade as a minimum replica count on a managed endpoint — rather than a faster start.