Fixing ModelError on a SageMaker Endpoint

An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (500) from primary with message — and then, usually, something unhelpfully truncated. The important thing about this error is that it is not SageMaker’s error. It is your container’s error, wrapped.

What ModelError means

The InvokeEndpoint API reference defines it in one sentence: “Model (owned by the customer in the container) returned 4xx or 5xx error code.” The HTTP status of the ModelError response itself is 424 — Failed Dependency, which is precisely the right code and tells you exactly where to look. SageMaker successfully routed your request, the container received it, and the container answered with an error.

So none of the usual endpoint checks help. The endpoint is InService, the IAM permissions are fine, the instance is healthy. What failed is code inside the container, and everything below is about getting to that code’s own message.

The three fields in the exception

AWS documents ModelError as carrying three fields, and most people never see two of them because the default string repr of the exception does not show them:

• OriginalStatusCode — the status your container returned. This is the single most diagnostic field, because 400 and 500 have completely different causes.
• OriginalMessage — the body your container returned, often truncated, sometimes containing the actual Python traceback.
• LogStreamArn — the ARN of the CloudWatch log stream for the instance that served the request. On a multi-instance endpoint this saves you reading the wrong stream.

In boto3 they arrive in the response dictionary on the ClientError. Print them rather than the exception:

from botocore.exceptions import ClientError

try:
    resp = rt.invoke_endpoint(
        EndpointName="my-endpoint",
        ContentType="application/json",
        Body=json.dumps(payload),
    )
except ClientError as e:
    err = e.response["Error"]
    if err["Code"] == "ModelError":
        print("status  :", e.response.get("OriginalStatusCode"))
        print("message :", e.response.get("OriginalMessage"))
        print("logs    :", e.response.get("LogStreamArn"))
    raise

Log all three at the call site permanently, not just while debugging. A ModelError caught and re-raised without them is an incident you will have to reproduce to investigate.

Getting to the log that explains it

If LogStreamArn is absent or you are looking at an error from an hour ago, go to the log group directly. AWS documents endpoint logs at /aws/sagemaker/Endpoints/[EndpointName], with streams named [production-variant-name]/[instance-id]. Async endpoints add a .../data-log stream, and inference pipelines add a stream per container.

aws logs tail /aws/sagemaker/Endpoints/my-endpoint \
  --since 30m --follow --format short

Everything your container writes to stdout or stderr lands here, which means the actual traceback is here even when OriginalMessage truncated it. Two things to look for beyond the obvious exception:

• The request that preceded the failure. Most serving containers log each invocation, so the last successful line before the traceback tells you what the failing input looked like.
• Whether the worker restarted. A container that dies and is restarted between requests produces intermittent ModelError responses that look load-related but are not. Cross-check MemoryUtilization for the same window — an out-of-memory kill is the usual cause and it does not always leave a Python traceback behind.

The causes, by original status code

Split on OriginalStatusCode first. It halves the search space.

A 4xx from the container means the request was wrong. Almost always a payload the container’s input function could not parse. The specific pattern that catches people is a mismatch between the ContentType you sent and what the container was built to accept: a container expecting text/csv given application/json will reject a perfectly valid payload. The other frequent 4xx is a schema mismatch — the right content type but the wrong keys, or the right keys with a nested shape the container’s deserialiser does not handle. Reproduce it by invoking with the container’s own documented example payload; if that works and yours does not, the difference is your bug.

A 5xx from the container means the code broke. The usual list, in rough order of frequency: an unhandled exception in the inference function, most often a shape or dtype mismatch on the first real input; an out-of-memory kill on a payload larger than anything tested; a missing file the container expected to find in the model artifact, which fails on the first request rather than at load time if the load is lazy; and a dependency that imports at request time and is not installed. All four are visible in the log stream as a traceback.

One 5xx cause is not a bug in your code: the request exceeded the container’s response window. AWS documents that a model container must respond within 60 seconds. Work that reliably takes longer belongs on asynchronous inference, where the timeout is a per-request parameter rather than a fixed ceiling. If the same payload succeeds when small and fails when large, suspect this before suspecting the model.

Three errors this is confused with

• ModelNotReadyException (HTTP 429). Not an error in your container at all. AWS documents it as meaning a serverless variant’s resources are still being provisioned, or a multi-model endpoint is still downloading or loading the target model. The documented response is to wait and retry. Do not route this into your throttling logic.
• ValidationError (HTTP 400). SageMaker rejected the request before it reached your container — a body over the 6,291,456-byte limit, an endpoint name that does not exist, an endpoint not in service. Nothing in your container logs will mention it, because nothing in your container saw it.
• ServiceUnavailable (HTTP 503) and InternalFailure (HTTP 500). These are SageMaker’s, not yours. Retry with backoff. If they persist rather than appearing intermittently, that is a support case, not a code change.

The discipline that makes all of this quick is branching on the error code rather than on the message text. ModelError goes to your container logs, ModelNotReadyException goes to a short bounded retry, ValidationError goes to the caller as a 400, and the 5xx family goes to backoff. Written once at the client, that turns a class of three-hour investigations into a log line that names the right place to look.