Building a Multi-Model Endpoint on SageMaker

A multi-model endpoint is an ordinary SageMaker endpoint with two fields changed and one large behavioural consequence. The fields take five minutes. The consequence — that some fraction of your requests will wait for a model to be downloaded from S3 and loaded into a container before they get an answer — is the whole design problem, and it is what this page is mostly about.

What actually changes in the API

Exactly two things, both in the container definition you pass to CreateModel. You set Mode to MultiModel, and you point ModelDataUrl at an S3 prefix rather than at a single model.tar.gz. Everything downstream — CreateEndpointConfig, CreateEndpoint, the production variant, the instance type — is identical to a single-model endpoint. Amazon documents both changes on its create a multi-model endpoint page.

The consequence of the second change is the one people miss. Because the endpoint is given a prefix and not a manifest, it does not know what models exist. It discovers them at invocation time. That is why adding a model is just an S3 upload with no endpoint update, why deleting one is an S3 delete, and why a typo in the model name at invoke time is a runtime error rather than a deploy-time one.

SageMaker AI describes the loading lifecycle on its multi-model endpoints page: on a request for a model that is not resident, it downloads the artifact from S3 to the instance’s storage volume, then loads it into the container’s memory. When memory runs short it unloads the least-used models — they stay on the storage volume, so a later re-load skips the download — and when the storage volume fills, it deletes unused artifacts from disk too. So there are three tiers, not two: in container memory, on local disk, in S3. Each miss costs more than the last.

Building the endpoint

Lay the artifacts out under one prefix first. Each one is a normal tarball for whatever serving container you are using, and the key relative to the prefix is the name you will pass at invoke time.

s3://my-bucket/models/
  tenant-a/scorer.tar.gz
  tenant-b/scorer.tar.gz
  tenant-c/scorer.tar.gz

1. Define the container with the two changed fields. The image must be one that supports multi-model mode; AWS keeps the supported list on its multi-model support page, and for GPU-backed endpoints it must be an NVIDIA Triton Inference Server image.

   import boto3
   
   sm = boto3.client("sagemaker")
   
   container = {
       "Image": image_uri,
       "ModelDataUrl": "s3://my-bucket/models/",
       "Mode": "MultiModel",
   }

2. Create the model, then the endpoint config, then the endpoint. AWS recommends at least two instances so the endpoint spans Availability Zones; on a multi-model endpoint that also raises the aggregate cache hit rate, because each instance keeps its own set of loaded models.

   sm.create_model(
       ModelName="tenant-scorers",
       ExecutionRoleArn=role_arn,
       Containers=[container],
   )
   
   sm.create_endpoint_config(
       EndpointConfigName="tenant-scorers-cfg",
       ProductionVariants=[{
           "VariantName": "AllTraffic",
           "ModelName": "tenant-scorers",
           "InstanceType": "ml.m5.xlarge",
           "InitialInstanceCount": 2,
           "InitialVariantWeight": 1,
       }],
   )
   
   sm.create_endpoint(
       EndpointName="tenant-scorers",
       EndpointConfigName="tenant-scorers-cfg",
   )

3. Wait for the endpoint to reach InService. Use the waiter rather than a sleep loop, because endpoint creation time depends on the instance type and the image size.

   sm.get_waiter("endpoint_in_service").wait(EndpointName="tenant-scorers")

There is a third, optional field worth knowing about. MultiModelConfig takes a ModelCacheSetting which is Enabled by default; setting it to Disabled makes the container unload each model after use. That sounds like a bad idea and usually is, but it is the right answer when your artifacts are large enough that two of them cannot coexist in memory, because thrashing an over-subscribed cache is worse than never caching at all.

Invoking one model out of many

One extra parameter on InvokeEndpoint: TargetModel, carried on the wire as the X-Amzn-SageMaker-Target-Modelheader. Its value is the artifact’s key relative to the prefix.

rt = boto3.client("sagemaker-runtime")

resp = rt.invoke_endpoint(
    EndpointName="tenant-scorers",
    TargetModel="tenant-b/scorer.tar.gz",
    ContentType="application/json",
    Body=json.dumps(payload),
)
print(resp["Body"].read())

Nothing else about the call changes. The 6,291,456-byte request and response body limits documented on the InvokeEndpoint API reference still apply, and so does the rule on that same page that a model container must respond within 60 seconds — which now includes the time spent downloading and loading your model.

The cold-load penalty, measured properly

The failure that surprises people is not slowness. It is a 429. AWS documents ModelNotReadyException on InvokeEndpoint as covering the case where “a multi-model endpoint is still downloading or loading the target model”, and its instruction is to wait and retry. If your client treats 429 as a rate limit and backs off for a minute, a first-call-to-a-cold-model looks like throttling. It is not; it is a cache miss with an HTTP status code. Handle it with a short, bounded retry rather than your throttle policy.

The rest is visible in CloudWatch, and AWS publishes five metrics for exactly this on its multi-model endpoint metrics page. Read them in this order:

• ModelCacheHit in the AWS/SageMaker namespace. Taken as an Average it is the ratio of requests whose model was already loaded. This is the single number that says whether the endpoint is working as intended. If it is not close to 1, nothing else you tune will matter.
• ModelLoadingWaitTime, in microseconds, is what a caller actually felt: how long the request waited for a download, a load, or both. Watch its Max, not its Average — the Average is dominated by hits.
• ModelDownloadingTime and ModelLoadingTime split that wait into the S3 fetch and the container’s LoadModel call. A large download time means bigger instances with more disk or fewer, smaller artifacts; a large load time means the model itself is slow to initialise and no amount of disk helps.
• LoadedModelCount in /aws/sagemaker/Endpoints, emitted per instance. Sum across instances and compare with your model count: if the sum keeps climbing and then collapsing, you are watching eviction.

When the tradeoff stops paying

AWS is unusually direct about the shape this works in: models on the same framework and the same serving container, similar in size and latency, with a mix of frequently and infrequently accessed models, and an application that tolerates occasional cold-start latency. Each clause is a real constraint. One container means one framework, so a PyTorch model and an XGBoost model do not share an endpoint. Similar sizes matter because the cache is shared: one artifact several times the size of the others evicts several of them every time it loads.

The clause that actually decides it is the last. AWS’s own guidance is that models with significantly higher throughput or latency requirements belong on dedicated endpoints. A model serving steady traffic gets no benefit from this architecture at all — it will be resident permanently, so you have paid the complexity for nothing and inherited a cache that other tenants can disturb. The pattern earns its keep on a long tail: hundreds or thousands of models where all of them together fit in an instance count you would not consider running as separate endpoints.

One structural caveat, and it is why this page carries a refresh decay. SageMaker has since grown inference components, a separate mechanism for packing several models onto one endpoint with explicit per-model resource reservations and per-copy autoscaling, invoked with InferenceComponentName instead of TargetModel. It is not a rename of this feature — components are declared rather than discovered, and they do not evict each other — but it covers some of the same ground, and which one AWS steers you toward has been moving. Before building anything long-lived on Mode: MultiModel, read the current hosting overview.

Once the endpoint is up, the next two questions are how it scales and what it costs. Scaling has its own subtlety on multi-model endpoints, because adding an instance starts a fresh, empty cache — see autoscaling a SageMaker endpoint. And when a container does fail rather than merely stall, the response you get is a 424, covered in fixing ModelError.