context_length_exceeded: What It Means and Five Fixes

context_length_exceeded means the request you sent was longer than the model can hold, and the request was rejected before any generation happened. It arrives as an HTTP 400 with a machine-readable code in the error body, and the message almost always contains the two numbers you need to fix it.

Read the two numbers first

Wording differs between providers and changes between releases, but the body has a consistent shape: a 400 status, an error object, a type of roughly invalid_request_error, and a code of context_length_exceeded or a close relative such as string_above_max_length.

{
  "error": {
    "message": "This model's maximum context length is 128000 tokens.
                However, your messages resulted in 131204 tokens.
                Please reduce the length of the messages.",
    "type": "invalid_request_error",
    "param": "messages",
    "code": "context_length_exceeded"
  }
}

The first number is the window. The second is what your request measured once the provider tokenised it. Subtract them and you have the overflow — here 3,204 tokens, or about 2.5% of the window. That size tells you which fix to reach for and it is the single most useful thing in the message.

• An overflow of a few hundred tokens is a budgeting bug. Something grew slightly past a boundary you thought you were under. Do not restructure anything; find the estimate that is wrong.
• An overflow of a few thousand is usually one extra turn of conversation history or one extra retrieved document. A trim policy fixes it permanently.
• An overflow of multiples of the window — 420,000 tokens into a 128,000 window — is a whole document, a whole log file, or a loop that appended the same content repeatedly. Trimming will not save you; the design is wrong.

What actually counts towards the limit

The overflow is nearly always in a component people forget to count. Everything in this list occupies the same window:

If your own estimate said 90,000 and the provider said 131,204, the gap is almost certainly tool schemas, images, or the chat template overhead, in that order of likelihood. That gap has its own page: the same string measured by two tokenisers.

Which part overflowed

Do not guess. Measure each component separately, once, and you will normally find that one of them is 80% of the request. This harness takes whatever token-counting function you already have — a local tokeniser, a provider counting endpoint, or the previous response’s usage.prompt_tokens divided appropriately — and reports the breakdown.

import json

def breakdown(messages, tools, count):
    """count(text) -> int. Use the tokeniser that matches YOUR model."""
    rows = []
    for i, m in enumerate(messages):
        text = m.get("content") or ""
        if not isinstance(text, str):          # multimodal parts
            text = json.dumps(text)
        rows.append((f"{i}:{m['role']}", count(text)))
    if tools:
        rows.append(("tool schemas", count(json.dumps(tools))))
    total = sum(n for _, n in rows)
    rows.sort(key=lambda r: -r[1])
    for name, n in rows[:10]:
        print(f"{n:>8}  {100*n/total:5.1f}%  {name}")
    print(f"{total:>8}  100.0%  TOTAL (excludes template overhead)")

Run it against the exact payload you sent, not against a reconstruction. The commonest surprise is a single tool result — an API response, a database dump, a page of HTML — that was appended to the conversation verbatim and has been resent on every turn since.

The five fixes, in the order they work

1. Cap the conversation history. Keep the system message, the last N turns, and drop the middle. This is the cause perhaps half the time and it is a ten-line change. Dropping the middle rather than the start matters: the system message and the original task usually carry the constraints, while turn seven of forty rarely does. Where the middle is genuinely needed, summarise it into a single message rather than deleting it.
2. Truncate tool results before they enter the conversation. A tool that returns 40,000 tokens of JSON should return the fields the model needs. Cap every tool result at a fixed budget at the point where you append it, and put the full payload somewhere the model can ask for by ID.
3. Lower top-k or shrink chunks in retrieval. Ten chunks of 1,500 tokens is 15,000 tokens of context, and material in the middle of a long context is attended to less reliably anyway, so this fix frequently improves quality as well as fitting. Add a reranker and pass the top three rather than the top ten.
4. Cut the tool list. Schemas are paid on every request. If twenty tools are registered and any one task needs four, select the relevant subset per request. This also improves tool selection accuracy, for the same reason.
5. Move to a larger window, last. It is the fix that requires no thought and it is last on purpose: it costs more per request, it does not fix the growth that got you here, and a request that has grown once will grow again. Use it to buy time, not to close the ticket.

The variant that is not overflow at all

There is a second failure that reports the same code and has a completely different fix. Several providers require that prompt_tokens + max_tokens fits inside the window, because the output has to be generated into the same buffer. So a 130,000-token window with a 126,000-token prompt and max_tokens set to 8,192 fails — even though the prompt on its own is comfortably inside.

You can recognise it from the arithmetic: if the reported request size is exactly your prompt plus your max_tokens, this is what happened. The fix is to compute the ceiling rather than hard-coding it.

WINDOW = 128_000
SAFETY = 500            # chat template overhead you did not count

max_tokens = min(desired_output, WINDOW - prompt_tokens - SAFETY)
if max_tokens < 256:
    raise ValueError("prompt leaves no room for a useful answer")

Raising that ValueError rather than clamping to a tiny number is deliberate. A request that succeeds with max_tokens=40 returns a truncated answer with finish_reason: "length", which is a quieter and worse failure than a 400.

Making it impossible rather than rare

The durable fix is a budget enforced in code before the request leaves. Decide the split up front — system prompt and tools get a fixed allowance, retrieval gets a fixed allowance, history gets what is left minus the output reservation — and have the assembly function trim to that budget rather than hoping. A request that cannot exceed the window cannot return this error.

Then log usage.prompt_tokens from every response and alert on the 99th percentile crossing 80% of the window. The alert fires days before the error does, which is the difference between a scheduled change and a production incident. There is more on the budgeting side in allocating a context window across components.