Right-Sizing a SageMaker Instance Type for Cost

A real-time SageMaker endpoint bills per instance-hour for as long as it exists, so the instance type is the single largest lever on its cost. Choosing a smaller one is a measurement problem, and the measurements have two units in them that reliably send people the wrong way.

The metrics that decide it

SageMaker publishes endpoint telemetry across two CloudWatch namespaces, and knowing which lives where saves an hour. Invocation metrics are in AWS/SageMaker: Invocations, InvocationsPerInstance, ModelLatency, OverheadLatency, Invocation4XXErrors and Invocation5XXErrors. Host resource metrics are in /aws/sagemaker/Endpoints: CPUUtilization, MemoryUtilization, and on GPU instances GPUUtilization and GPUMemoryUtilization. AWS documents both namespaces here.

Four of those decide the sizing. Utilisation tells you how much of the instance you are using; InvocationsPerInstance tells you the throughput each instance is carrying; ModelLatency tells you whether the container is keeping up; and OverheadLatency separates the model’s own time from everything SageMaker does around it, which is how you tell a slow model from a slow endpoint.

aws cloudwatch get-metric-statistics \
  --namespace /aws/sagemaker/Endpoints \
  --metric-name CPUUtilization \
  --dimensions Name=EndpointName,Value=my-endpoint \
               Name=VariantName,Value=AllTraffic \
  --start-time 2026-07-14T00:00:00Z \
  --end-time   2026-08-11T00:00:00Z \
  --period 3600 \
  --statistics Average Maximum

Two units that mislead

The first is CPUUtilization. AWS documents it as the sum across all cores, so on a four-vCPU instance the range is 0 to 400% rather than 0 to 100. A reading of 150% on such an instance is not catastrophic overload; it is 37.5% of the box. Teams who read it as a percentage of the whole instance conclude they are saturated and upsize, which is exactly backwards. GPUUtilization behaves the same way per GPU: on a four-GPU instance the range is 0 to 400%.

The second is ModelLatency, which AWS documents as being recorded in microseconds. A dashboard that labels it milliseconds is off by a factor of a thousand in the direction that makes everything look fine, and the effect is that a 200 ms model reads as 200 µs and nobody investigates. Check the unit on every latency panel before trusting any of them.

Neither of these is a subtle inference; both are stated in the documentation. They are worth stating again because they are the two most common reasons a right-sizing exercise reaches a confidently wrong answer.

Turning utilization into a target

1. Pull at least two weeks of hourly Average and Maximum for the utilisation metric that binds — GPU utilisation for a GPU-served model, CPU otherwise — plus InvocationsPerInstance over the same window.
2. Normalise the utilisation figure by the core or GPU count so you are reasoning about a fraction of the instance, not a raw percentage.
3. Take the peak, not the mean. The instance has to survive the busiest hour of the fortnight, and a mean over a workload with a daily shape hides that hour completely.
4. Compare peak normalised utilisation to the ratio between your current instance and the candidate. Moving from an instance with four vCPUs to one with two roughly doubles utilisation, so a peak under about 35% of the current instance is the crude precondition for that move with headroom left over.
5. Check memory separately. Memory is a hard wall rather than a gradient: a model that fits in 16 GB and not in 8 does not run slower on the smaller instance, it fails to load.

The relationship between instance size and throughput is not linear for model serving. Batching behaviour, thread pool sizing inside the container, and memory bandwidth all bend it, which is why the arithmetic above is a filter for candidates rather than an answer.

Letting Inference Recommender search

Rather than reasoning about candidates, you can have SageMaker load test them. CreateInferenceRecommendationsJob runs your model across a set of instance types and returns, per candidate, InstanceType, InitialInstanceCount, tuned EnvironmentParameters, and the metrics MaxInvocations, ModelLatency, CostPerHour and CostPerInference. AWS documents a default job as taking up to about 45 minutes. The API reference is here.

CostPerInference is the number this whole exercise is chasing, and it is the one that most often contradicts intuition: a larger instance that serves proportionally more requests can have a lower cost per inference than a small one, and the cheapest hourly rate is frequently not the cheapest endpoint. Read that column before the hourly one.

Two limits worth knowing. The job itself provisions instances and is therefore not free, so treat it as a periodic exercise rather than a continuous one. And it load tests with traffic you supply, so the recommendation is only as representative as the sample payloads — feed it real requests, including the long ones.

Checking the SLA survives

Downsizing is applied by creating a new endpoint configuration and calling UpdateEndpoint, which performs a managed rollout rather than a stop and start. The endpoint keeps serving throughout, and you can roll back by updating to the previous configuration.

aws sagemaker create-endpoint-config \
  --endpoint-config-name my-endpoint-cfg-smaller \
  --production-variants '[{
      "VariantName": "AllTraffic",
      "ModelName": "my-model",
      "InitialInstanceCount": 2,
      "InstanceType": "ml.g5.xlarge"
  }]'

aws sagemaker update-endpoint \
  --endpoint-name my-endpoint \
  --endpoint-config-name my-endpoint-cfg-smaller

Then watch the right statistic. Average ModelLatency barely moves under moderate additional pressure; the p99 moves first, and it is the p99 that a downstream timeout is set against. Set a CloudWatch alarm on the p99 before the update, not after, so a breach is caught by alarm rather than by a support ticket.

Watch two more things for a full traffic cycle. Invocation5XXErrors rising after a downsize usually means the container is now hitting a memory or concurrency limit under load rather than failing outright. And OverheadLatency should be unaffected by instance size — if it moved, something other than the instance changed with it.

Finally, right-sizing is second-best to not running the endpoint at all. An endpoint with a handful of requests a day is a candidate for serverless inference or for deletion, not for a slightly smaller instance — the forgotten endpoint page is about that case.