Llama’s Output Length: No Hard API Cap, Just max_new_tokens

Hosted models publish two numbers: a context window and a maximum output length, and the second is usually much smaller than the first. Llama’s model cards publish only the first, and the omission is not an oversight — there is nobody in the architecture whose job it would be to enforce a second one.

The number that does not exist

A hosted provider caps output for reasons that are all about running a service. A single request generating for an hour occupies a scheduling slot and holds KV cache the whole time; capping it bounds the tail latency of everyone else’s requests and makes capacity planning possible. The cap is a policy of the service, and it appears in the documentation as a separate figure precisely because it is a separate decision from what the model can do.

When you download weights, no such policy exists. The checkpoint is a function from tokens to a distribution over the next token; there is no field in config.json for a maximum generation length, and no component that would consult one. The loop that calls the model repeatedly is yours, and it runs until you tell it to stop. That is the whole of the difference.

One consequence: if you are calling Llama through a hosted reseller and hitting an output cap, that cap is the reseller’s and belongs in their documentation, not Meta’s. Two hosts of the same weights can legitimately publish different maximum output lengths, and neither is more correct.

The context window is the real ceiling

There is one hard limit, and it is shared rather than separate. Prompt tokens and generated tokens live in the same sequence, so:

prompt_tokens + generated_tokens  <=  context_length

With Llama 3.1’s 131,072-token window and a 100,000-token prompt, the most you can generate is about 31,000 tokens regardless of what you set. With the original Llama 3’s 8,192-token window and a 7,000-token prompt, you have roughly 1,192 left. The available output length is therefore a function of your prompt, which is exactly the relationship hosted APIs hide by publishing a fixed number.

What happens when you exceed it depends on the runtime and none of the answers are good. Transformers will raise or produce garbage as positions run past the trained range; vLLM validates up front and rejects the request; llama.cpp may shift the context window, silently discarding the oldest tokens — which means the model loses your system prompt mid-answer and nothing errors. The error strings and their causes are in context length exceeded on a local model.

Remember also that the context you can actually use is the served length rather than the configured one, which is usually smaller. See Llama 3’s context window across releases.

The parameter, runtime by runtime

The same concept has four names, and two of them mean subtly different things:

Transformers   max_new_tokens=512      tokens to generate
Transformers   max_length=512          prompt + generated (avoid)
vLLM           max_tokens=512          tokens to generate
llama.cpp      -n 512 / --predict      tokens to generate
               -n -1                   unlimited
               -n -2                   until context is full
Ollama         num_predict: 512        tokens to generate

The Transformers pair is the trap. max_length counts the prompt; max_new_tokens does not. Set max_length=512 with a 400-token prompt and you get 112 tokens of output, which reads as a model that abruptly stops. Use max_new_tokens unless you specifically want the other behaviour, and note that passing both makes max_new_tokens win with a warning.

llama.cpp’s -n -1 is worth flagging separately as a production hazard. Unlimited generation on a model that fails to emit a stop token is an infinite loop that fills your context and your bill. Prefer -n -2 if you want “as much as fits”, because it terminates.

The default that catches everyone

In Hugging Face Transformers, generate() with no length argument falls back to max_length, whose default is 20 tokens. Not 20 words, and not a sentence — twenty tokens, which is a fragment.

# Truncated at 20 tokens including the prompt
out = model.generate(**inputs)

# What you meant
out = model.generate(**inputs, max_new_tokens=512)

The symptom is a model that answers with the first few words of a correct answer and stops, and it gets misdiagnosed as a broken checkpoint or a bad chat template roughly every time. A model card with a generation_config.json may override the default, so the behaviour differs between checkpoints, which makes it worse rather than better. Always pass the value explicitly.

Reserving room before you truncate

Because prompt and output share one budget, any code that trims a conversation to fit has to trim to the context length minus the output you intend to generate. Trimming to the full context length is a bug that only shows up on long conversations, which is the worst possible schedule for finding it.

SERVED_CONTEXT = 8192      # what the server allocated, not the config
MAX_OUTPUT     = 1024      # what you will pass as max_new_tokens
SAFETY         = 64        # template overhead: headers, injected dates

budget = SERVED_CONTEXT - MAX_OUTPUT - SAFETY   # 7104 tokens of prompt

def fit(messages, tok):
    """Drop whole turns from the oldest end until the rendered
    prompt fits. Never split a turn: half a message renders as a
    malformed template."""
    while True:
        n = len(tok.apply_chat_template(messages,
                                        add_generation_prompt=True))
        if n <= budget or len(messages) <= 2:
            return messages, n
        # keep index 0 if it is the system message
        drop = 1 if messages[0]["role"] == "system" else 0
        del messages[drop]

Three details in that are the ones that matter. Measure the rendered length, not the sum of message lengths — the template adds header tokens per message and, on 3.1 and later, injects date lines you did not write. Drop whole turns rather than truncating text, because a half-message leaves the format inconsistent in a way the model was never trained on. And preserve the system message, which is otherwise the first thing an oldest-first policy discards, silently removing your instructions from a long conversation.

The SAFETY margin is not superstition. If the injected Today Date line changes length, or a tool schema is added, or your token count came from a different tokenizer version, you want the request to still fit rather than to fail at the boundary.

Why long answers stop anyway

Raising the parameter does not reliably produce a longer answer, because the parameter is a ceiling and not a target. The model stops when it emits a stop token, and instruct tuning gives it strong opinions about how long a reply should be — opinions formed from the length distribution of its fine-tuning data, not from your parameter.

• The model decided it was done. Check the finish reason. If generation ended on <|eot_id|> rather than on the token limit, the length is the model’s choice and the fix is in the prompt: ask for a specific structure, a number of sections, or a minimum length.
• Quality degrades before the limit does. Coherence over very long generations falls off well inside the context window, and repetition loops become more likely the longer generation runs. Several shorter calls with explicit structure beat one very long one for most work.
• Time scales linearly. Output is generated one token per forward pass, so a 4,000-token answer takes roughly eight times as long as a 500-token one. Raising the cap raises your worst-case latency by the same factor, which is the reason hosted providers impose the cap you are no longer subject to.
• It may not have stopped at all. If output runs to the limit every time and ends mid-sentence, suspect stop-token configuration rather than model verbosity — see Llama 3’s two end tokens.