502, 503 and 504 From an AI Endpoint

A 502 Bad Gateway, 503 Service Unavailable or 504 Gateway Timeout from a model API was generated by one of four hops, and only one of them is the provider’s inference service. Reading the body and two headers tells you which, and that determines whether the fix is yours or a wait.

Four hops, any of which can answer

1. Your own egress. A corporate proxy, a service mesh sidecar, an API gateway you run. Frequently the source of 502s on long streaming responses, because its read timeout is shorter than the generation.
2. The provider’s edge. A CDN or load balancer. Produces 502 and 504 when its backend misbehaves, and its errors usually look nothing like the provider’s documented error format.
3. The provider’s API layer. Authentication, rate limiting, routing. Its errors are JSON in the format the documentation describes.
4. The inference worker. Where the model runs. Overload here typically surfaces as 503 with a retry hint, or as a 504 from the layer in front of it.

Whose error is it

curl -sS -D- -o /tmp/body https://api.example-provider.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d @request.json
head -c 400 /tmp/body      # HTML or JSON? that is the attribution

Capture and log the first few hundred bytes of every 5xx body along with the status. An error class that discards the body in favour of a tidy message throws away the only evidence that distinguishes these cases, and it is the commonest reason a 5xx investigation stalls.

What each code actually means here

• 502 — an intermediary got an invalid response from its upstream, or the upstream closed the connection mid-response. On model APIs the classic cause is a long or streaming response crossing a proxy that gave up. If your 502s correlate with long generations, this is it, and streaming is the fix rather than a longer timeout.
• 503 — capacity. The service is up and declining work: a busy model, a deployment in progress, a regional issue. This is the one that is genuinely worth retrying, and it is the one most likely to carry Retry-After. If it persists for a specific model while others work, it is that model’s capacity rather than the platform.
• 504 — a gateway waited and gave up. The shortest timeout anywhere in the chain wins, so a 504 tells you about a deadline, not about a failure. Something was still working when the connection was cut.
• 500 — an unhandled error inside the application. Occasionally provoked by your request: an enormous payload, invalid UTF-8, a parameter combination that slipped past validation. If a single specific request reproduces a 500 every time and others do not, it is your payload, and bisecting it is faster than waiting for a status page.

Note what is not in this list. A 429 is rate limiting, which is a different mechanism with a different remedy — handling 429 and rate limits cover it. Treating them together is how a rate-limit problem gets a capacity fix.

The round number that proves a timeout

This is the most useful diagnostic on the page and it takes one query over data you probably already have. Plot the time-to-failure of your 5xx responses. Genuine faults fail at scattered times. Timeouts fail at a suspiciously exact value.

SELECT status,
       round(duration_ms / 1000.0) AS seconds,
       count(*)
FROM llm_requests
WHERE status >= 500 AND ts > now() - interval '7 days'
GROUP BY 1, 2
ORDER BY 3 DESC
LIMIT 20;

A pile of failures at 30, 60 or 100 seconds is somebody’s configured deadline, and the round value usually identifies the component: 30 and 60 seconds are common defaults in load balancers and serverless platforms, 100 seconds is a well-known default elsewhere. Find the component with that number and you have found the cause. The fixes, in order: stream the response so bytes flow before the deadline expires; raise the specific timeout you identified; or make the work smaller.

A related pattern worth checking at the same time: 5xx concentrated on large request bodies is a payload-size limit, which some proxies express as a 502 rather than the correct 413.

When only some requests fail

A 5xx rate of 100% is a straightforward outage. A rate of two or ten percent is more informative, because whatever distinguishes the failing requests from the successful ones is the cause. Group the failures by every dimension you record and look for one that is not uniform.

SELECT model, route, streaming,
       width_bucket(prompt_tokens, 0, 200000, 10) AS size_bucket,
       count(*) FILTER (WHERE status >= 500) AS failures,
       count(*)                              AS total,
       round(100.0 * count(*) FILTER (WHERE status >= 500) / count(*), 2) AS pct
FROM llm_requests
WHERE ts > now() - interval '24 hours'
GROUP BY 1, 2, 3, 4
HAVING count(*) > 50
ORDER BY pct DESC;

• Concentrated in large prompts. A body-size limit or an upload timeout somewhere in the chain. Both are configuration, and both are usually on a component you own.
• Concentrated in streaming requests. An intermediary that cannot hold the connection open. Same territory as a hung stream.
• Concentrated in one model. That model’s capacity, not the platform. Failing over to another model is available; waiting for the platform is not the same decision.
• Uniform across everything, in bursts. A genuine upstream incident, or your own instance restarting. Check whether the bursts align with your deploys before assuming it is theirs.
• Concentrated on one of your instances. One unhealthy host: an exhausted connection pool, a leaked file descriptor, a stale DNS entry. Group by host as well; this is the dimension people forget to record.

Retrying without making it worse

All three of these are retryable in principle, and a naive retry loop is a reliable way to turn a provider’s brief degradation into your own outage.

RETRYABLE = {429, 500, 502, 503, 504}

def backoff(attempt, retry_after=None):
    if retry_after is not None:
        return float(retry_after)                 # honour it exactly
    return min(2 ** attempt, 30) * (0.5 + random.random())   # full jitter

• Cap the attempts at three or four. An unbounded retry on a long generation multiplies both the load and the bill; retries are a recurring entry in bill triage for exactly this reason.
• Jitter is not optional. Without it, every client that failed together retries together, and the second wave is larger than the first.
• Do not retry a 504 with the same long request. It will take just as long and fail the same way. Change something: stream, shorten, or use a different route.
• Add a circuit breaker. After a threshold of consecutive failures, stop calling and fail fast for a cooldown. This protects the provider and stops your own request queue backing up — circuit breakers and safe retries cover the mechanics.
• Be careful with non-idempotent work. A retried request that already had an effect — a tool call that sent an email, a charge, a write — is worse than a failure. Use an idempotency key where the provider supports one.