Deploying a Container to a SageMaker Real-Time Endpoint

A SageMaker real-time endpoint is three API calls over one container image, and the calls are the easy part. What decides whether the endpoint reaches InService is whether your image answers two HTTP routes fast enough, and the failure when it does not is a fifteen-minute wait followed by a rollback.

The container contract

SageMaker does not care what is inside your image. It requires that the container listens on port 8080 and serves two routes:

• GET /ping — health check. Return 200 once the model is loaded and able to serve. This is the one that matters: SageMaker polls it after starting the container and will not put the endpoint in service until it succeeds.
• POST /invocations — inference. The request body is whatever your caller sent, with the Content-Type they set; you return the response body and its content type.

The most common deployment failure is a /ping that returns 200 immediately while a large model is still loading in the background. SageMaker marks the container healthy, routes a request to it, the request fails, and you get a ModelError that has nothing to do with your model code. Load first, then answer /ping — a boolean flag set at the end of load is the whole fix.

# A minimal contract, FastAPI
from fastapi import FastAPI, Request, Response

app = FastAPI()
ready = False

@app.on_event("startup")
def load():
    global model, ready
    model = load_from("/opt/ml/model")   # where SageMaker unpacks ModelDataUrl
    ready = True

@app.get("/ping")
def ping():
    return Response(status_code=200 if ready else 503)

@app.post("/invocations")
async def invocations(request: Request):
    payload = await request.json()
    return {"prediction": model.predict(payload["inputs"])}

Note /opt/ml/model. If you set ModelDataUrl on the model, SageMaker downloads and extracts that archive there before starting your container. Baking weights into the image instead is faster to start and slower to iterate on; pulling them from S3 at runtime yourself is the worst of both, because you pay the download on every scale-out and SageMaker cannot see it happening.

CreateModel

The model resource binds an image to an execution role and, optionally, artefacts and environment variables.

aws sagemaker create-model \
  --model-name summariser-v3 \
  --execution-role-arn arn:aws:iam::111122223333:role/SageMakerExecutionRole \
  --primary-container '{
    "Image": "111122223333.dkr.ecr.us-east-1.amazonaws.com/summariser:3.1.0",
    "ModelDataUrl": "s3://acme-models/summariser/3.1.0/model.tar.gz",
    "Environment": {"MODEL_PRECISION": "bf16", "MAX_BATCH_SIZE": "8"}
  }'

Pin the image by immutable tag or by digest. :latest here means a scale-out event six weeks from now silently pulls a different model onto half your fleet, and nothing in the endpoint’s state will tell you that happened. The execution role needs ecr:GetDownloadUrlForLayer and ecr:BatchGetImage on the repository plus s3:GetObject on the artefacts.

CreateEndpointConfig, and the two timeouts

The endpoint configuration is where instance type, count and the startup timeouts live. It is immutable once created, so a change means a new config and an UpdateEndpoint — which is also what makes blue/green possible.

aws sagemaker create-endpoint-config \
  --endpoint-config-name summariser-v3-ml-g5-xlarge \
  --production-variants '[{
    "VariantName": "primary",
    "ModelName": "summariser-v3",
    "InstanceType": "ml.g5.xlarge",
    "InitialInstanceCount": 2,
    "InitialVariantWeight": 1.0,
    "ModelDataDownloadTimeoutInSeconds": 1800,
    "ContainerStartupHealthCheckTimeoutInSeconds": 1800
  }]'

Those last two fields are the ones to set deliberately for anything large. ModelDataDownloadTimeoutInSeconds bounds pulling and extracting ModelDataUrl; a multi-gigabyte archive can exceed a short default and the endpoint fails with a message about the container not starting, which sends you to debug the container. ContainerStartupHealthCheckTimeoutInSeconds bounds how long SageMaker waits for the first successful /ping — this is the budget your model-loading time spends. Raise both before assuming your image is broken.

InitialVariantWeight only matters with more than one variant, and two weighted variants on one config is how you do a canary: deploy the new model as a second variant at weight 0.05, watch its metrics, then shift weight. InitialInstanceCount of 1 means every deployment and every instance replacement is an outage, so 2 is the real minimum for anything that matters.

CreateEndpoint

1. aws sagemaker create-endpoint --endpoint-name summariser --endpoint-config-name summariser-v3-ml-g5-xlarge.
2. Poll describe-endpoint until EndpointStatus leaves Creating. The states are Creating, InService, Updating, RollingBack, Deleting and Failed.
3. If it fails, read FailureReason on the describe response first, then the CloudWatch log group /aws/sagemaker/Endpoints/summariser. The failure reason is usually about health checks; the log stream contains the exception that caused them.
4. Deploy the next version by creating a new endpoint config and calling update-endpoint. SageMaker brings up the new fleet before draining the old one, so the endpoint name and its callers do not change.
5. Configure autoscaling separately — it is Application Auto Scaling against the SageMakerVariantInvocationsPerInstance metric, not a field on the endpoint. See endpoint autoscaling.

UpdateEndpoint takes an optional DeploymentConfig, and it is worth setting rather than accepting the default. A BlueGreenUpdatePolicy lets you shift traffic in canary or linear steps with a bake period between them, and AutoRollbackConfiguration takes a list of CloudWatch alarms that abort the deployment and restore the previous fleet if any of them fires during the bake. Without it, a model that loads successfully and answers badly is a fully successful deployment as far as SageMaker is concerned — the health check only knows whether /ping returned 200. Point the alarms at your own error rate and latency metrics, because those are the only signals that distinguish a working deployment from a running one.

The quota that blocks CreateEndpoint

The most common way a first deployment fails has nothing to do with the container. SageMaker maintains its own per-instance-type quotas for endpoint usage, separate from EC2 limits and separate again from the quotas for training and processing jobs on the same instance type. On a new account the endpoint quota for GPU families is frequently zero, so a perfectly valid CreateEndpoint fails immediately:

An error occurred (ResourceLimitExceeded) when calling the CreateEndpoint
operation: The account-level service limit 'ml.g5.xlarge for endpoint usage'
is 0 Instances, with current utilization of 0 Instances and a request delta
of 2 Instances. Please use AWS Service Quotas to request an increase for
this quota.

The useful part of that message is the quoted quota name. It is a literal Service Quotas entry, one per instance type, and the suffix is what people get wrong when they raise a ticket: raising ml.g5.xlarge for training job usage does nothing for an endpoint, and the two are approved by different teams on different timescales. Read the name out of the error and request exactly that.

• It is per Region. An increase granted in us-east-1 does not apply in eu-west-1, and discovering that during a multi-Region rollout is a bad afternoon.
• It counts instances, not endpoints. InitialInstanceCount of 2 needs a quota of at least 2, and a blue/green update needs headroom for both fleets at once — so an endpoint running at exactly its quota cannot be updated without downtime.
• GPU increases are not instant. Treat the request as a lead-time item in the project rather than something to discover on launch day.

Invoking it

The endpoint is not a public URL. It is a SageMaker Runtime API call signed with SigV4, which means IAM controls access and there is no key to leak.

import boto3, json

rt = boto3.client("sagemaker-runtime", region_name="us-east-1")

response = rt.invoke_endpoint(
    EndpointName="summariser",
    ContentType="application/json",
    Accept="application/json",
    Body=json.dumps({"inputs": "Summarise the following ticket: ..."}),
)

print(json.loads(response["Body"].read()))

Two documented limits shape everything you can do with it. The request and response bodies are each capped at 6,291,456 bytes — 6 MB. And AWS states that a model container must respond within 60 seconds, that the model itself has a maximum processing time of 60 seconds, and that if you are running at 50 to 60 seconds you should set the SDK socket timeout to 70.

That 60-second ceiling is the fact to design around, because a generative model producing a long answer will exceed it. The documented answers are asynchronous inference for long jobs, or streaming via InvokeEndpointWithResponseStream so tokens leave the container as they are produced rather than in one response at the end.

When inference fails, the exception is ModelError at HTTP 424 — meaning your container returned a 4xx or 5xx. It carries OriginalStatusCode, OriginalMessage and, most usefully, LogStreamArn pointing straight at the CloudWatch stream for that instance. Log that ARN in your error handler and the next ModelError takes one click to diagnose instead of a log search.