Async Inference on SageMaker for Long-Running Requests

Asynchronous inference exists because of one hard number: a SageMaker model container must respond to a synchronous invocation within 60 seconds. If your inference does not, no amount of client-side timeout tuning will save it, and async is the supported way out.

The ceiling you are escaping

The InvokeEndpoint API reference states it plainly: customer model containers must respond within 60 seconds, the model itself has a maximum processing time of 60 seconds, and if you expect to use 50–60 of them your SDK socket timeout should be 70. There is also a hard payload ceiling — request and response bodies are capped at 6,291,456 bytes.

Asynchronous inference lifts both. Payloads move through S3 rather than through the request body, and AWS documents an invocation timeout you can set as high as 3,600 seconds. In exchange you give up the synchronous reply: the API returns HTTP 202 with the location where the answer will eventually appear.

Configuring the endpoint

The model is unchanged. The difference is entirely in CreateEndpointConfig, which gains an AsyncInferenceConfig. Per the AsyncInferenceConfig reference, it holds a required OutputConfig and an optional ClientConfig. The output config carries S3OutputPath, S3FailurePath, an optional KmsKeyId, and a NotificationConfig.

1. Create the endpoint config with an output path and, ideally, a distinct failure path. Splitting them is worth doing: without S3FailurePath you will be inspecting objects to find out whether each one is an answer or a stack trace.

   sm.create_endpoint_config(
       EndpointConfigName="doc-extract-async-cfg",
       ProductionVariants=[{
           "VariantName": "AllTraffic",
           "ModelName": "doc-extract",
           "InstanceType": "ml.g5.xlarge",
           "InitialInstanceCount": 1,
       }],
       AsyncInferenceConfig={
           "OutputConfig": {
               "S3OutputPath": "s3://my-bucket/async/out/",
               "S3FailurePath": "s3://my-bucket/async/fail/",
               "NotificationConfig": {
                   "SuccessTopic": success_topic_arn,
                   "ErrorTopic": error_topic_arn,
               },
           },
           "ClientConfig": {
               "MaxConcurrentInvocationsPerInstance": 4,
           },
       },
   )

2. Create the endpoint from that config and wait for InService exactly as you would for a real-time endpoint.
3. Give the execution role read access to the input prefix and write access to both output prefixes. This is the step that fails silently-ish: the invocation succeeds with a 202 and nothing ever appears at the output location.

MaxConcurrentInvocationsPerInstance is the throttle between the queue and your container. Set it to the number of simultaneous requests one instance can genuinely hold. Set it too high and the queue stops protecting the container, which is the entire reason you are here.

Submitting and collecting a job

Upload the payload, call invoke_endpoint_async with InputLocation, and hold on to what comes back. The InvokeEndpointAsync reference documents an HTTP 202 carrying OutputLocation and FailureLocation headers plus an InferenceId in the body.

import boto3, json

s3 = boto3.client("s3")
rt = boto3.client("sagemaker-runtime")

s3.put_object(Bucket="my-bucket", Key="async/in/job-1.json",
              Body=json.dumps(payload).encode())

resp = rt.invoke_endpoint_async(
    EndpointName="doc-extract-async",
    InputLocation="s3://my-bucket/async/in/job-1.json",
    ContentType="application/json",
    InferenceId="job-1",
    InvocationTimeoutSeconds=1800,
)
print(resp["OutputLocation"], resp["FailureLocation"])

There is a shortcut for small payloads that is easy to miss: Body accepts an inline payload of up to 128,000 bytes, and is mutually exclusive with InputLocation. If your input is small but your processing is long, you can skip the S3 round trip entirely and still get the async execution model.

Collecting the result means polling S3 for the output key, or subscribing to the SNS topics. Prefer the topics. The polling version is written more often and is worse in a specific way: it usually polls only OutputLocation, so a failed job looks identical to a slow one until the poll loop gives up. If you do poll, poll both locations.

from botocore.exceptions import ClientError

def collect(bucket, out_key, fail_key):
    for _ in range(120):
        for key, kind in ((out_key, "ok"), (fail_key, "failed")):
            try:
                obj = s3.get_object(Bucket=bucket, Key=key)
                return kind, obj["Body"].read()
            except ClientError as e:
                if e.response["Error"]["Code"] not in ("NoSuchKey", "404"):
                    raise
        time.sleep(5)
    return "timeout", None

The two timeouts that govern the request

These are separate, they mean different things, and their defaults are not the numbers people assume.

• InvocationTimeoutSeconds is how long the request may spend being processed before it is marked expired. AWS documents the default as 900 seconds and the maximum as 3,600. The important half of that is the default: if you moved to async because a job takes twenty minutes, and you did not set this field, the job still expires.
• RequestTTLSeconds is how long the request may sit in the queue before it is marked expired — documented default 21,600 seconds (6 hours), minimum 60, maximum 21,600. This one is a backlog policy. On a queue that has fallen badly behind, a long TTL means you eventually process work whose answer nobody wants any more.

Async endpoints also get their own log stream layout. AWS documents the endpoint log group as /aws/sagemaker/Endpoints/[EndpointName], with async endpoints adding a [production-variant-name]/[instance-id]/data-log stream alongside the usual container stream. When a job fails and the failure object is unhelpful, that data log is the next place to look.

Why your usual dashboard is empty

This is the part that surprises people a week in, and it is worth knowing before you build anything on top. AWS states on its alarms and logs for asynchronous endpoints page that its list of async metrics is exhaustive, and that any metric not on it is not published for an async-enabled endpoint. The examples it gives are Invocations, InvocationsPerInstance and OverheadLatency.

So the standard endpoint dashboard goes blank, and more importantly the standard autoscaling policy stops being available. The predefined SageMakerVariantInvocationsPerInstance metric type used for real-time endpoint autoscaling is built on a metric that async endpoints do not emit. That is not a preference for backlog-based scaling; it is why the customised metric in the next section is the only option rather than the recommended one.

What you get instead is better, once you know the names. Four metrics carry only the EndpointName dimension and describe the queue: ApproximateBacklogSize (items queued or in flight), ApproximateBacklogSizePerInstance (the same divided by instance count, which AWS says is primarily for autoscaling), ApproximateAgeOfOldestRequest in seconds, and HasBacklogWithoutCapacity. Alarm on the third of those, not the first: a backlog of 500 is meaningless without knowing whether it drains in a minute or has been stuck for an hour, and age is the number your users actually experience.

The rest carry EndpointName and VariantName, and they split a request’s lifetime into the pieces you would otherwise guess at. TimeInBacklog is queue time only, explicitly excluding download, upload and model latency. TotalProcessingTime is receipt to completion including all of those. Subtract one from the other and compare the remainder with ModelLatency, and you know whether a slow job was slow because the model was slow or because it waited — which decides whether you add instances or optimise the container. RequestDownloadLatency and ResponseUploadLatency account for the S3 legs separately, and they are where an unexpectedly large payload shows up.

The failure counters are the ones to put alarms on, because each names a distinct cause that would otherwise present as “the output never appeared”:

• RequestDownloadFailures — SageMaker could not read your input object. Almost always the execution role, or a KMS key it cannot use.
• ResponseUploadFailures — the inference succeeded and the answer could not be written. This is the cruel one: you paid for the work and have nothing to show, and the container logs look perfectly healthy.
• ExpiredRequests — requests that died in the queue on RequestTTLSeconds. A non-zero value here means the endpoint is structurally under-provisioned, not that a job was faulty.
• NotificationFailures — the work finished but the SNS publish failed, so a notification-driven consumer never fires. Without this alarm the pipeline simply stops and nothing reports an error.
• InvocationFailures and InvocationsProcesssed for the overall ratio.

Two smaller things worth knowing while you are here. The SNS notification can carry the inference response itself rather than just a pointer, via IncludeInferenceResponseIn — an array of at most two values drawn from SUCCESS_NOTIFICATION_TOPIC and ERROR_NOTIFICATION_TOPIC — but AWS documents that the response is included only if it is 128 KB or smaller, so a consumer built on it must still handle the pointer case. And the data-log stream contains each request’s inference ID, which is what lets you map a failure back to a specific job; pass your own InferenceId at submission time and that mapping becomes your own job identifier rather than a generated one.

One structural point that follows from all of this: the container never touches S3. SageMaker downloads the input object and POSTs it to the container’s /invocations endpoint exactly as it would for a real-time request, then uploads whatever comes back. The container contract is unchanged, which means an existing real-time container works as-is — and it means writing S3-reading code into the container, which is a common first instinct, is both unnecessary and the reason the download metrics stop matching reality.

Scaling to zero, and waking up again

Async endpoints are the one hosting option where AWS supports scaling instances down to zero, which is why they are also a cost answer and not only a duration answer. Registration is the same as a real-time endpoint except that MinCapacity is 0, and the tracked metric is a customised one: ApproximateBacklogSizePerInstance in the AWS/SageMaker namespace, dimensioned on EndpointName.

There is a trap here that AWS documents but which is easy to skip past. At zero instances, a backlog-per-instance metric divides by zero — so the target-tracking policy alone will not bring the endpoint back. AWS’s autoscale an asynchronous endpoint page therefore describes a second, step-scaling policy driven by a CloudWatch alarm on the HasBacklogWithoutCapacity metric, which adds one instance when the queue is non-empty and capacity is zero.

aas.put_scaling_policy(
    PolicyName="HasBacklogWithoutCapacity-ScalingPolicy",
    ServiceNamespace="sagemaker",
    ResourceId=resource_id,
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    PolicyType="StepScaling",
    StepScalingPolicyConfiguration={
        "AdjustmentType": "ChangeInCapacity",
        "MetricAggregationType": "Average",
        "Cooldown": 300,
        "StepAdjustments": [
            {"MetricIntervalLowerBound": 0, "ScalingAdjustment": 1}
        ],
    },
)

Without that second policy the endpoint still recovers eventually — but only once the backlog exceeds the target-tracking value, which on a low-traffic endpoint can mean the first request of the day waits a very long time. If your traffic is sporadic enough that this matters, compare against serverless inference, which trades the long-duration ceiling for genuinely per-request billing.