Gemma's Output Token Limit and Default Generation Length

Searching for Gemma’s maximum output tokens usually returns a number from someone’s hosted endpoint. The weights themselves do not carry a per-size output ceiling, and knowing where the real bound comes from is what stops your answers being cut off at a length nobody chose.

The only ceiling the weights impose

A causal language model generates until something stops it. There are exactly three stopping conditions: it emits a stop token, the caller hits a token limit, or the sequence reaches the position limit the model was built with. Only the third is imposed by the checkpoint, and it is shared with the prompt:

max_new_tokens <= max_position_embeddings - len(prompt_tokens)

On Gemma 2, max_position_embeddings is 8,192 at every size, so a 6,000-token prompt leaves at most 2,192 tokens of answer. On Gemma 3’s 4B and larger the ceiling is 128K, and on the Gemma 3 1B it is 32K. There is no separate output budget in either generation, which is the substantive difference between an open-weight model and a hosted API that publishes an output cap alongside a context length.

Hosted APIs publish a separate output number for two reasons that do not apply to a checkpoint you run yourself. One is operational: a request that generates for ten minutes occupies a slot, so capping completion length caps worst-case request duration. The other is predictability for the caller, who would rather have a bounded bill than an unbounded answer. Neither is a statement about what the model can do, which is why the same weights carry different advertised limits at different providers.

The default that truncates at 20 tokens

The most common Gemma truncation has nothing to do with Gemma. If you call Hugging Face transformers’ generate() without specifying a length, the library’s own default applies, and it is very short: 20 new tokens. People report the model “stopping mid-sentence” and go looking for a stop-token bug that is not there.

out = model.generate(**inputs)                      # ~20 tokens. Almost never what you want.
out = model.generate(**inputs, max_new_tokens=512)  # explicit, and the right habit

Always pass max_new_tokens rather than max_length. max_length counts the prompt, so the same value produces different answer lengths for different prompts, and a long prompt can leave no room at all for output while looking generous.

The second common truncation is a serving framework’s own default. vLLM, llama.cpp, Ollama and Text Generation Inference each ship a default completion length and each picks a different one. If answers are consistently cut at a suspiciously round number, that number is in your server config, not in the weights.

Distinguishing a length cut from a stop-token cut takes one field. Every sensible inference API reports why generation ended, and the value is what tells you which knob to reach for: a length reason means your limit bound, a stop reason means the model finished on its own. In transformers there is no such field, so the equivalent check is whether the last generated id is in the stop set:

stop_ids = model.generation_config.eos_token_id
stop_ids = stop_ids if isinstance(stop_ids, list) else [stop_ids]
print("stopped naturally:", int(out[0][-1]) in stop_ids)

If that prints False, you were truncated and raising max_new_tokens is the fix. If it prints True and the answer still looks short, the model chose to stop, and the fix is in the prompt.

What the checkpoint’s generation config says

Gemma checkpoints carry a generation_config.json, and it is worth reading because it also holds the stop-token set that decides when generation ends on its own:

from transformers import GenerationConfig

gc = GenerationConfig.from_pretrained("google/gemma-2-9b-it")
print(gc.eos_token_id)      # note: often a LIST, not a single id
print(gc.bos_token_id)
print(gc.max_length)        # if present, a default rather than a hard cap

The eos_token_id being a list is the detail that matters most. Gemma instruction-tuned checkpoints stop on <end_of_turn> as well as on <eos>, and a runtime that only honours a single stop id will run past the end of the turn and keep writing. That failure looks like an output-limit problem and is not one; the stop-token page takes it apart.

Hosted endpoints impose their own cap

Gemma is served by Google’s own API, by Vertex AI, and by a long list of third-party providers, and each of those publishes a maximum completion length that is a property of the service. Two providers serving byte-identical weights can legitimately advertise different output caps, because the cap is an operational decision about request duration rather than a fact about the model.

The practical rule: when you read a Gemma output limit, ask whose limit it is. For the open weights the answer is derived from the window. For a service, the answer is in that service’s model listing and can change without the weights changing at all. Google publishes its own limits alongside the model documentation at ai.google.dev/gemma/docs.

There is usually a request timeout sitting behind the token cap, and it binds first more often than people expect. A small model generating at, say, forty tokens a second under load needs the best part of a minute for a 2,000-token answer, and a sixty-second gateway timeout will cut that off with a transport error rather than a clean stop reason. If long answers fail intermittently while short ones never do, look at the timeout before the token limit.

Streaming changes the calculation rather than removing it. Streaming does not make generation faster, but it moves the timeout risk from total request duration to the gap between chunks, which is short and stable. On any endpoint where you expect long answers, streaming is the difference between an intermittent failure at the transport layer and a request that simply takes as long as it takes.

Budgeting output against the window

Because prompt and completion share one budget, reserve the completion first and let the prompt take what is left. The inverse order silently produces truncated answers on your longest, most important requests:

WINDOW    = 8192          # Gemma 2, every size
RESERVED  = 1024          # answer length you are willing to pay for

prompt_ids = tok.apply_chat_template(messages, tokenize=True,
                                     add_generation_prompt=True)
if len(prompt_ids) > WINDOW - RESERVED:
    raise ValueError(f"prompt is {len(prompt_ids)} tokens; trim to {WINDOW - RESERVED}")

out = model.generate(input_ids=..., max_new_tokens=RESERVED)

Raising rather than trimming automatically is the safer default in a pipeline. Silent truncation of a prompt drops whichever end your slicing happened to cut, and on a folded system instruction that end is often the instruction itself.

Where you do have to trim, trim the middle of the conversation rather than either end. The oldest turns hold the folded instruction and the newest hold the actual question, so dropping turns from the middle outwards preserves both. That is more code than slicing, and it is the difference between a long conversation that degrades gracefully and one that abruptly forgets its own rules.

One last note on cost, since output length is where most of the money goes. Generation is one forward pass per token, so a 1,000-token answer takes roughly ten times as long as a 100-token one on the same hardware, while the prompt is processed in a single parallel pass. On a small model served locally, the reserved output length is the dominant term in your latency budget. Setting it generously “just in case” costs nothing when answers are short and everything when the model decides to be thorough.