A Terraform Module for an ECS GPU Task

The GPU part of an ECS task definition is four lines. Everything that makes those four lines work is somewhere else: in the AMI, in the agent configuration, in the container image and in the capacity provider. A module is worth writing precisely because it holds that chain together.

The chain a GPU task depends on

AWS documents four separate requirements for GPU workloads on ECS, and missing any one of them produces a task that is accepted and never runs.

• A supported instance family. AWS lists p2, p3, p4d, p5, g3, g4, g5, g6, g6e and g6f as the GPU-based container instance types providing access to NVIDIA GPUs, and notes that g2 support is deprecated and p2 works only on GPU-optimized AMI versions earlier than 20230912.
• The ECS GPU-optimized AMI, which ships pre-configured NVIDIA kernel drivers and a Docker GPU runtime. A plain ECS-optimized AMI has neither.
• ECS_ENABLE_GPU_SUPPORT set to true in the agent configuration on the container instance. This is a user-data edit, not a Terraform resource.
• NVIDIA_DRIVER_CAPABILITIES in the image for containers not built on an NVIDIA or CUDA base image. AWS documents that it must be set to utility,compute or all, and that ECS sets NVIDIA_VISIBLE_DEVICES for you but does not set the others.

One more constraint that decides the whole architecture: AWS states that GPUs are not supported on Windows containers, and the GPU runtime, drivers and AMI all live on the container instance. That is why a GPU task is an EC2 launch-type task — requires_compatibilities = ["EC2"] — and why the Fargate Spot batch pattern is a different design rather than a variation on this one.

Capacity provider and launch template

Resolve the AMI rather than hard-coding it. AWS publishes the GPU-optimized AMI ID in Systems Manager Parameter Store, which means the module never carries a stale ID:

data "aws_ssm_parameter" "ecs_gpu_ami" {
  name = "/aws/service/ecs/optimized-ami/amazon-linux-2/gpu/recommended/image_id"
}

resource "aws_launch_template" "gpu" {
  name_prefix   = "${var.name}-gpu-"
  image_id      = data.aws_ssm_parameter.ecs_gpu_ami.value
  instance_type = var.instance_type   # e.g. "g5.xlarge"

  iam_instance_profile { arn = aws_iam_instance_profile.ecs_instance.arn }

  user_data = base64encode(<<-EOT
    #!/bin/bash
    echo "ECS_CLUSTER=${var.cluster_name}" >> /etc/ecs/ecs.config
    echo "ECS_ENABLE_GPU_SUPPORT=true"     >> /etc/ecs/ecs.config
  EOT
  )
}

resource "aws_ecs_capacity_provider" "gpu" {
  name = "${var.name}-gpu"

  auto_scaling_group_provider {
    auto_scaling_group_arn         = aws_autoscaling_group.gpu.arn
    managed_termination_protection = "ENABLED"

    managed_scaling {
      status                    = "ENABLED"
      target_capacity           = 100
      minimum_scaling_step_size = 1
      maximum_scaling_step_size = 2
    }
  }
}

target_capacity = 100 means ECS aims for no spare instance — the right default for GPU hardware, where an idle instance is the expensive kind of idle. It also means a new task waits for an instance to boot, which on a GPU AMI is not fast. If your traffic is bursty and latency-sensitive, trade it down knowingly rather than leaving the default.

managed_termination_protection requires the Auto Scaling group to have instance protection enabled and scale-in protection set; without it the capacity provider will refuse to create. This is a real ordering dependency between two resources that look independent.

The task definition

resource "aws_ecs_task_definition" "this" {
  family                   = var.name
  requires_compatibilities = ["EC2"]
  network_mode             = "awsvpc"
  cpu                      = var.cpu
  memory                   = var.memory
  execution_role_arn       = aws_iam_role.execution.arn
  task_role_arn            = aws_iam_role.task.arn

  container_definitions = jsonencode([
    {
      name      = var.name
      image     = var.image
      essential = true

      resourceRequirements = [
        {
          type  = "GPU"
          value = tostring(var.gpu_count)
        }
      ]

      environment = [
        { name = "NVIDIA_DRIVER_CAPABILITIES", value = "utility,compute" }
      ]

      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = aws_cloudwatch_log_group.this.name
          "awslogs-region"        = data.aws_region.current.name
          "awslogs-stream-prefix" = "ecs"
        }
      }
    }
  ])
}

resourceRequirements lives inside the container definition, not at task level, and the value is a string even though it is a count. Both of those have produced long-running provider issues; if the field appears to be ignored, check that it is nested inside the container object and that the number is quoted.

Reserving GPUs here is what makes ECS pin physical devices to your container. AWS documents the opposite mode too: to share a GPU between containers you remove the GPU resource requirement entirely and make nvidia the default Docker runtime in instance user data, then set NVIDIA_VISIBLE_DEVICES yourself. That is a different module with a different contract, and mixing the two on one cluster is how you get non-deterministic placement.

The service and its placement

resource "aws_ecs_service" "this" {
  name            = var.name
  cluster         = var.cluster_arn
  task_definition = aws_ecs_task_definition.this.arn
  desired_count   = var.desired_count

  capacity_provider_strategy {
    capacity_provider = aws_ecs_capacity_provider.gpu.name
    weight            = 1
    base              = 0
  }

  network_configuration {
    subnets         = var.subnet_ids
    security_groups = [aws_security_group.task.id]
  }

  placement_constraints {
    type       = "memberOf"
    expression = "attribute:ecs.instance-type == ${var.instance_type}"
  }
}

The placement constraint is optional but usually correct. AWS shows the same expression form in its GPU documentation, and on a mixed cluster it is what stops a task that needs one L4 from landing on an eight-GPU instance and stranding seven of them. Clusters may contain a mix of GPU and non-GPU container instances, which is exactly why the constraint earns its place.

What updates and what replaces

Three resources in this module have completely different update semantics, and the difference decides whether a plan is a deployment, a no-op, or a change that quietly does not reach your running machines.

The task definition never updates. ECS task definitions are immutable, so any change — a new image tag, one more environment variable, a different GPU count — produces a new revision with a new ARN rather than modifying the existing one. Because the service above references aws_ecs_task_definition.this.arn, and that ARN carries the revision number, a task definition change necessarily updates the service, which is a deployment. That is usually what you want, but it means an apply that looks like a one-line edit rolls your GPU tasks. Terraform also never deletes old revisions; they accumulate indefinitely, which is harmless but makes the console’s revision list useless for archaeology after a few months.

The launch template updates without touching anything. This is the trap worth remembering. Changing image_id — which happens on its own, because the SSM parameter resolves to whatever AMI AWS currently recommends — or instance_type creates a new launch template version. The Auto Scaling group adopts it for new instances only. Your running GPU instances keep the old AMI, and therefore the old NVIDIA driver, until something replaces them. On a pool that rarely scales, that is indefinitely.

If the AMI change matters — and for a driver-level change it does — make the replacement explicit rather than waiting for a scale event:

resource "aws_autoscaling_group" "gpu" {
  # ...
  launch_template {
    id      = aws_launch_template.gpu.id
    version = aws_launch_template.gpu.latest_version
  }

  instance_refresh {
    strategy = "Rolling"

    preferences {
      min_healthy_percentage = 50
      instance_warmup        = 300
      skip_matching          = true
    }
  }
}

Rolling is the only strategy this argument accepts. instance_warmup deserves a real number here rather than the default: a GPU instance has to boot, register with the cluster and pull a CUDA-sized image before it is genuinely ready, and declaring it healthy early means the refresh terminates the next instance while the replacement is still pulling. skip_matching stops the refresh replacing instances that already run the current template, which makes re-running it cheap.

The service argues with whoever edited the console. The predictable version of this is desired_count. Somebody scales the service up during an incident, and the next apply — by anyone, for any unrelated reason — silently scales it back to the number in the configuration. If an application autoscaling policy owns that number, say so, and let Terraform own only its initial value:

lifecycle {
  ignore_changes = [desired_count]
}

Use ignore_changes narrowly and only where another system is the legitimate owner of the field. Applying it to the whole resource to silence a noisy plan converts real drift into invisible drift, which on GPU capacity means an instance type you did not choose, running until somebody reads the bill.

When the task never starts

The symptom is a service event reading roughly unable to place a task because no container instance met all of its requirements, often followed by a note about the GPU resource. Work down the chain in this order:

1. Is there a registered container instance at all? Scaling from zero on GPU capacity can fail on stock-out, which surfaces as an Auto Scaling activity error rather than an ECS one.
2. Does the instance report GPUs? Describe the container instance and look for GPU in its registered resources. If it reports none, the AMI or ECS_ENABLE_GPU_SUPPORT is wrong — those two produce identical symptoms.
3. Are the GPUs already reserved? ECS pins whole devices, so on a single-GPU instance a second task requesting one GPU is unplaceable no matter how much CPU and memory is free.
4. Does the placement constraint match a real instance type? A typo here produces a permanently pending task with no GPU-specific error at all.

The equivalent problem on Kubernetes has a different shape and a different diagnostic path — see provisioning a GKE GPU node pool for how driver installation is expressed there. And whichever platform you are on, do not edit GPU capacity without state locking.