SageMaker Serverless Inference: Setup and Its Real Limits

Serverless inference scales to zero between requests and bills by the millisecond, which sounds like the obvious default for an intermittent workload. For language models it usually is not, and the reason is a single line in the feature exclusion list rather than anything to do with cost.

Read the exclusions first

AWS documents a specific list of real-time features that serverless endpoints do not support. It is worth reading before anything else, because most of the questions people bring to this page are answered by it:

• GPUs. Serverless is CPU-only. That alone rules it out for hosting most open-weight language models at usable latency, and it is the single most common reason a plan to use it collapses.
• VPC configuration and network isolation. If the model must reach a private database, or your compliance posture requires the endpoint inside a VPC, serverless cannot do it.
• Multi-model endpoints, inference pipelines and multiple production variants. No canary by variant weight, and no hosting fifty small models behind one endpoint.
• Data capture and Model Monitor. Drift monitoring has to be built yourself.
• AWS Marketplace model packages and private Docker registries. The image has to be one you can pull from ECR.

One more, which is a one-way door: AWS states that you cannot convert an instance-based real-time endpoint to serverless — attempting the update returns a ValidationError — while you can convert serverless to real-time and cannot roll that back. Decide before you create.

The image itself is capped at 10 GB. AWS also recommends creating only one worker in the container and loading one copy of the model, unlike real-time containers that often fork a worker per vCPU. A container tuned for a real-time endpoint will over-allocate memory on a serverless one and fail in a way that looks like a model size problem.

Six memory sizes, and what they buy

There is no instance type. There is a memory size, and AWS documents exactly six values: 1024, 2048, 3072, 4096, 5120 and 6144 MB — minimum 1 GB, maximum 6 GB. Compute is assigned proportionally, so a larger memory selection also means more vCPU. You cannot buy CPU without buying memory.

AWS’s guidance is that memory size should be at least as large as your model, and each increment is priced differently. Regardless of which you pick, the endpoint gets 5 GB of ephemeral disk.

The practical consequence of a 6 GB ceiling is worth stating plainly: it bounds what can be hosted here at all. A model whose weights do not fit in 6 GB alongside its runtime cannot use serverless inference, full stop, and no amount of quantisation config changes the ceiling. Classical models, small transformers, embedding models and rerankers fit comfortably; a general-purpose chat model does not.

The concurrency arithmetic

Three separate numbers govern capacity, and AWS documents all three:

• Per endpoint: maximum concurrency up to 200.
• Per Region, per account: total concurrency shared across all serverless endpoints is 1000 in US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Europe (Frankfurt) and Europe (Ireland); 500 in the other supported Regions.
• Endpoint count: 50 serverless endpoints per Region.

The per-endpoint cap exists so one endpoint cannot consume the whole account pool, and AWS states that invocations beyond an endpoint’s maximum are throttled. So with a 1000 account pool and a 200 per-endpoint cap, five saturated endpoints exhaust the Region — the sixth endpoint gets nothing regardless of its own setting. If you are planning a dozen serverless endpoints, do that sum before you plan the twelfth.

Creating one

The model and endpoint resources are identical to a real-time deployment. Only the production variant changes: ServerlessConfig replaces InstanceType and InitialInstanceCount.

1. Build an image that answers GET /ping and POST /invocations on port 8080, with one worker and one copy of the model. Push it to ECR — a private registry will not work here.
2. create-model exactly as for real-time, naming the image and an execution role.
3. Create the endpoint config with a serverless variant:

aws sagemaker create-endpoint-config \
  --endpoint-config-name reranker-serverless \
  --production-variants '[{
    "VariantName": "primary",
    "ModelName": "reranker-v2",
    "ServerlessConfig": {
      "MemorySizeInMB": 4096,
      "MaxConcurrency": 20
    }
  }]'

1. create-endpoint against that config, then poll describe-endpoint for InService.
2. Invoke it with the same sagemaker-runtime invoke-endpoint call as a real-time endpoint. The client code does not change at all — which is the genuinely nice part of this service.
3. Add ProvisionedConcurrency to ServerlessConfig if you need warm capacity. AWS requires it to be less than or equal to the endpoint’s MaxConcurrency, and it can be autoscaled on a metric or a schedule through Application Auto Scaling — though AWS notes that autoscaling of provisioned concurrency is not supported in CloudFormation.

Cold starts, and when real-time wins

A serverless endpoint that has had no traffic must start compute before it can answer, and AWS notes a cold start can also occur when concurrent requests exceed current usage — so scaling up mid-burst costs the same penalty as scaling from zero. Duration depends on model size, download time and container start-up.

The number to measure is the CloudWatch metric OverheadLatency, which AWS documents as tracking the time to launch new compute for the endpoint. That is the metric that answers “is serverless viable for us” — not p50 latency, which hides the whole problem, but the p99 of OverheadLatency against your actual traffic gaps.

Where the economics turn over is straightforward once you frame it as duty cycle rather than request count. Serverless bills compute by the millisecond only while a request is in flight; real-time bills the instance by the second for as long as it exists. So serverless wins when the endpoint is idle most of the time and the workload tolerates a cold start, and loses when the endpoint is busy enough that per-millisecond billing approaches a permanently-running instance — at which point you are paying serverless rates for real-time utilisation and getting no GPU, no VPC and no monitoring for it.

Note also that provisioned concurrency undoes the main advantage: you are now paying for warm capacity by the duration provisioned, which is a rectangle, not an area. Once provisioned concurrency is on all day, compare it honestly against the cost of a small real-time endpoint before assuming serverless is still the cheaper shape.

One last operational detail: ModelNotReadyException, HTTP 429, is documented as meaning a serverless variant’s resources are still being provisioned. It is a retryable start-up condition, not a rate limit, and a client that treats every 429 as throttling and backs off for a minute will make your cold start dramatically worse.