Autoscaling a SageMaker Endpoint

SageMaker endpoints do not scale themselves. Scaling is done by Application Auto Scaling, a separate service that knows nothing about models, and the whole configuration is three API calls against a resource ID you have to construct by hand.

Autoscaling is three API calls

Register a scalable target, define a policy, apply it. The first two are the ones with sharp edges. It matters that this is Application Auto Scaling and not SageMaker, because it explains why the permissions are unrelated to your endpoint’s permissions and why the resource is addressed by a string rather than an ARN.

The caller needs sagemaker:DescribeEndpoint, sagemaker:DescribeEndpointConfig, sagemaker:UpdateEndpointWeightsAndCapacities, the application-autoscaling actions, the CloudWatch alarm actions, and iam:CreateServiceLinkedRole for AWSServiceRoleForApplicationAutoScaling_SageMakerEndpoint. That last one is the one people are missing when registration fails with a permissions error that names no obvious resource; AWS lists the full policy on its auto scaling prerequisites page.

Registering the scalable target

The resource ID is endpoint/NAME/variant/VARIANT, and the scalable dimension is sagemaker:variant:DesiredInstanceCount. Both are literal strings; the variant name is the one you set in ProductionVariants, which is AllTraffic if you took the convention from the SDK and something else if you did not.

1. Register the variant, choosing the bounds you are prepared to pay for. MinCapacity is a floor you will pay for continuously on a real-time endpoint, so pick it as a capacity decision rather than a safety margin.

   import boto3
   
   aas = boto3.client("application-autoscaling")
   resource_id = "endpoint/tenant-scorers/variant/AllTraffic"
   
   aas.register_scalable_target(
       ServiceNamespace="sagemaker",
       ResourceId=resource_id,
       ScalableDimension="sagemaker:variant:DesiredInstanceCount",
       MinCapacity=2,
       MaxCapacity=10,
   )

2. Apply a target-tracking policy on the predefined metric. AWS’s own example on the define a scaling policy page keeps average invocations per instance at 70.

   aas.put_scaling_policy(
       PolicyName="invocations-per-instance",
       ServiceNamespace="sagemaker",
       ResourceId=resource_id,
       ScalableDimension="sagemaker:variant:DesiredInstanceCount",
       PolicyType="TargetTrackingScaling",
       TargetTrackingScalingPolicyConfiguration={
           "TargetValue": 70.0,
           "PredefinedMetricSpecification": {
               "PredefinedMetricType": "SageMakerVariantInvocationsPerInstance"
           },
           "ScaleInCooldown": 600,
           "ScaleOutCooldown": 300,
       },
   )

3. Confirm the alarms exist. Application Auto Scaling creates a pair of CloudWatch alarms behind the policy, and looking at them is the fastest way to find out whether the policy is doing anything.

   aws cloudwatch describe-alarms \
     --alarm-name-prefix TargetTracking-endpoint/tenant-scorers

The target-tracking policy

The number 70 is not a throughput figure and reading it as one is the most common mistake here. InvocationsPerInstance is a count of invocations per instance per minute. A target of 70 therefore means roughly 1.2 requests per second per instance — which is either wildly conservative or wildly aggressive depending entirely on how long one of your inferences takes. Derive it instead: decide the concurrency one instance can hold without its p95 degrading, multiply by 60, divide by your mean inference seconds. If an instance comfortably serves 4 concurrent requests at 2 seconds each, that is about 120 invocations per minute, and a target somewhere below that leaves headroom for the scale-out to complete.

Target tracking on this metric is genuinely the right default for steady request-response traffic. It is a poor default for generation workloads, for the reason in the next section.

Why it reacts slowly, and what to use instead

Two independent lags stack up. First, AWS documents that standard CloudWatch metrics including InvocationsPerInstance emit once every minute. Second, target tracking works through CloudWatch alarms, which need several consecutive datapoints before they fire. Add the time it takes SageMaker to provision an instance and pull your container, and the interval between traffic arriving and capacity existing is measured in minutes, not seconds.

That is tolerable when each request takes 50 milliseconds and a queue drains as fast as it fills. It is not tolerable when each request occupies an instance for thirty seconds, because a burst does not queue — it saturates, and the metric that would tell you so is an invocation count, which does not rise when requests are long. An endpoint fully occupied by ten slow generations and an endpoint idling can report similar invocation counts.

AWS added predefined metrics for exactly this problem. SageMakerVariantConcurrentRequestsPerModelHighResolution tracks concurrent requests, counts requests that are queued inside the container, and — per AWS’s documentation — for models that stream tokens it tracks each request until the last token is sent. It emits every 10 seconds rather than every 60. The equivalent for inference components is SageMakerInferenceComponentConcurrentRequestsPerCopyHighResolution. If you are hosting anything generative, start here rather than with invocation counts.

TargetTrackingScalingPolicyConfiguration={
    "TargetValue": 5.0,
    "PredefinedMetricSpecification": {
        "PredefinedMetricType":
            "SageMakerVariantConcurrentRequestsPerModelHighResolution"
    },
}

A third option, for anything CPU-bound rather than concurrency-bound, is a customised metric on CPUUtilization in the /aws/sagemaker/Endpoints namespace with the EndpointName and VariantName dimensions. Note the range: AWS documents CPUUtilization as the sum across cores, so a four-core instance ranges 0–400%, and a target of 50 on a four-core instance means something very different from what you probably meant.

Cooldowns and the asymmetry you want

ScaleOutCooldown and ScaleInCooldown are optional and you should set both. AWS’s worked example uses 300 seconds out and 600 seconds in, and that asymmetry is the correct instinct generalised: being slow to add capacity costs latency now, being slow to remove it costs money later, and the second is the cheaper mistake.

On a multi-model endpoint the asymmetry matters more than usual. A new instance arrives with an empty model cache, so its first requests each pay a full download and load before answering. Scaling out under load therefore makes latency briefly worse before it makes it better, and scaling in throws away a warm cache you paid to build. Longer cooldowns on both sides, and a higher MinCapacity than you would otherwise choose, are the usual answer.

Finally, be careful about attaching more than one policy to a target. AWS warns that with multiple policies the largest capacity wins for both scale-out and scale-in, which means a policy you added as a safety net can quietly become the only one that ever decides anything. If the endpoint is idle for most of the day, the question is not what to scale it to but whether it should be a real-time endpoint at all — see why a SageMaker endpoint costs more than expected.