A Field Guide to LLM API Error Messages

Inference APIs return a small, stable set of failures, and most integrations handle them with a blanket retry that makes two of them worse and hides a third. Knowing which is which takes about ten minutes and saves an outage.

The shape of an error

Both major dialects return a JSON body with a structured error object alongside the HTTP status. In the OpenAI dialect it is {"error": {"message", "type", "param", "code"}}; Anthropic returns {"type": "error", "error": {"type", "message"}}. The status tells you the class; the type or code field tells you what to do, and it is the field most client code discards. Log both, and log the request id header — every provider issues one, and it is the only thing a support conversation can proceed from.

The distinction that organises everything below is not client-versus-server, which is what the status code nominally encodes. It is will the identical request succeed later? Three answers exist: yes after a wait (capacity and rate conditions), no until something changes in the request (validation, auth, model identity), and no until something changes outside the request entirely (a billing state, a retired snapshot, a regional restriction). Only the first is retryable, the second belongs in an alert on your own deploy, and the third needs a human. Several genuinely different conditions share a status code across that boundary, which is why classifying on status alone produces a retry policy that is wrong in both directions — hammering a wall in one place and giving up on a transient blip in another.

Error messages themselves are prose written for a human and are the worst thing to branch on. They get reworded without notice, they are sometimes localised, and the same underlying condition is phrased differently by two providers. Match on the status and the type field, keep the message for the log, and if you must string-match — some providers put the only useful detail in the message — treat that branch as a known liability and cover it with a test that runs against the live API rather than a fixture.

4xx: you have to change something

None of these is retryable. A retry loop on a 400 is a loop that runs until your timeout budget is gone, and a retry loop on a 401 during a key rotation is a self-inflicted outage.

429 is three different problems

This is where the blanket retry does real damage, because three distinct conditions share one status code.

• Request-rate limit. Too many requests per minute. Genuinely transient, respects Retry-After, and correctly handled by backoff. The x-ratelimit-remaining-requests and x-ratelimit-reset-requests headers let you avoid it proactively rather than discovering it.
• Token-rate limit. Too many tokens per minute, which is a different bucket with its own headers. A workload of few large requests hits this while the request counter looks healthy — confusing until you know the two buckets exist. Backoff works; reducing per-request size works better.
• Quota exhausted. The insufficient_quota condition is a billing state, not a rate limit. It will not clear on its own, and retrying it wastes your budget of attempts and can trigger further limiting. This must be detected on the error type, not on the status, and it should page someone rather than retry.

A fourth variant appears on some providers: a concurrency limit on in-flight requests, which needs a semaphore in your client rather than a backoff. If your 429 rate is insensitive to how long you wait, that is what you have.

5xx: retry, carefully

The billing detail that catches people: if a generation completed server-side and your client gave up, you may still be charged for it. Retrying a timeout can therefore mean paying twice for one answer, which is an argument for streaming and for generous client timeouts rather than aggressive ones.

Failures that arrive with a 200

• A stream that ends without a terminal event. The HTTP response succeeded and the content is incomplete. Covered on truncated output, and it is the most commonly unhandled failure in this whole guide.
• An error object inside the stream. Some providers open the stream, then emit an error event mid-flight — most often for a capacity failure or a content filter. Your SSE parser must handle an event type it was not expecting rather than ignoring it.
• A content filter as a stop reason. Status 200, partial text, and the reason for the truncation is in finish_reason.
• A refusal as content. A model declining a request is a successful response containing a refusal. Detect and route it — see false refusals — rather than treating it as an answer.
• A tool call with malformed arguments. Valid response, invalid JSON in the arguments field. Handle it as a model error with a corrective turn, not as an API error.

A retry policy worth copying

import random, time

RETRYABLE_STATUS = {408, 429, 500, 502, 503, 504, 529}
NEVER_RETRY_TYPES = {"insufficient_quota", "invalid_request_error",
                     "authentication_error", "permission_error",
                     "context_length_exceeded"}

def call_with_retry(fn, attempts=4, base=0.5, cap=20.0):
    for i in range(attempts):
        try:
            return fn()
        except ApiError as e:
            if e.type in NEVER_RETRY_TYPES or e.status not in RETRYABLE_STATUS:
                raise                                  # do not burn attempts
            if i == attempts - 1:
                raise
            # Honour Retry-After when present; otherwise exponential backoff
            # with FULL jitter -- equal jitter still synchronises a fleet.
            wait = float(e.retry_after) if e.retry_after else \
                   random.uniform(0, min(cap, base * (2 ** i)))
            time.sleep(wait)

Full jitter rather than a fixed multiplier is the detail that matters at scale: without it, every client that hit the same limit retries at the same moment and rebuilds the spike that caused the limit. Cap the total attempts rather than the delay alone, because an unbounded retry on a capacity event turns a degraded provider into a hung application.

Above the retry layer, one more decision: after the attempts are exhausted, fail over or fail? A 529 or a sustained 503 is a fleet condition, so a second model is likely to succeed where a fourth retry will not. A 400 is yours and failing over just moves it. Make that distinction explicit in code rather than leaving it to a timeout.