Phi-4's Context Window and Output Limit

Phi-4 has a 16K context window, which is shorter than the Phi-3 checkpoint it replaced. It has no documented output limit at all, and that is not an omission — it is what open weights mean.

The documented number

Microsoft’s model card for Phi-4 on Hugging Face states a context length of 16K tokens for the 14B dense model, and the accompanying technical report, “Phi-4 Technical Report” published in December 2024, describes the model as pretrained at a shorter length and extended to 16K during a midtraining stage. You can read the same figure off the checkpoint:

from transformers import AutoConfig
cfg = AutoConfig.from_pretrained("microsoft/phi-4")
print(cfg.max_position_embeddings)   # the declared context length
print(cfg.num_hidden_layers, cfg.hidden_size)

That is the only number the model itself commits to. Everything else people quote as a “Phi-4 limit” comes from the thing serving it.

Why it went down from Phi-3

Phi-3-medium shipped a 128K variant; Phi-4, its successor at a similar size, is documented at 16K. That looks like a regression and is better understood as a different bet. Long-context extension is a separate training stage with its own cost, and it is not free in quality — the rescaling that makes position 100,000 representable also perturbs the short-range positional signal the model spends most of its time using.

Phi’s stated thesis has always been data quality over scale, and Phi-4’s report emphasises synthetic data generation and post-training over architectural reach. A 16K window is enough for the reasoning and coding tasks the model was optimised against, and the long-context work reappears elsewhere in the family rather than in the flagship. If you need 128K from a small Microsoft model, the Phi-3.5 and Phi-4-mini lines are where to look — see the Phi-3 context window table.

There is no separate output limit

Hosted APIs publish two numbers: a context window and a maximum completion length. The second exists because the provider decided it does — it is a policy about how long one request may occupy a GPU. Phi-4 is a weights file. There is no provider in the loop and no policy, so there is no such number.

What constrains output is the window arithmetic. Prompt tokens and generated tokens share the same 16K. If your prompt is 12,000 tokens, the model has fewer than 4,400 left before positions run past what it was tuned for. Beyond that the failure is not an error — the runtime will keep sampling and the output will drift.

The cap you actually hit is a parameter you set, and it has a different name in every runtime:

transformers   model.generate(..., max_new_tokens=1024)
vLLM           SamplingParams(max_tokens=1024)
llama.cpp      ./llama-cli -n 1024        (n_predict)
Ollama         "options": { "num_predict": 1024 }
OpenAI-shaped  {"max_tokens": 1024}   posted to /v1/chat/completions

Note the trap in the first two lines: max_new_tokens counts only generated tokens, while some older interfaces used max_length, which counts prompt plus generation. Same intent, different arithmetic, and mixing them up produces one-token-long answers on long prompts.

Several runtimes also default this value low. If your Phi-4 deployment truncates at a suspiciously round number, the model is not stopping — your server is.

What happens when you go over

There is no single answer, and that is the problem. Each runtime handles an over-length request differently, and two of the behaviours are silent:

• An OpenAI-compatible server rejects it. vLLM returns HTTP 400 with a message naming both numbers — the tokens requested and the model’s maximum — which is the behaviour you want, because it is loud and it tells you the margin.
• llama.cpp shifts the context. By default it can discard tokens from the start of the sequence to make room and continue generating. Nothing fails. The model simply stops being able to see the beginning of your prompt, which on a retrieval-augmented request means it loses the system instructions first.
• A tokenizer truncates. Calling a tokenizer with truncation=True and a max length cuts the input to fit and returns it without complaint. If the truncation happens before the chat template is applied, you can lose the generation prompt and get a base-model continuation.
• Nothing at all happens, and the output degrades. Feeding positions past what the weights were tuned for is not an error condition in the arithmetic. The forward pass completes and produces fluent text that has stopped tracking the prompt.

Because the worst case is the quiet one, count before you send rather than relying on the runtime to object. One tokenizer call on the rendered conversation is far cheaper than the request itself.

Budgeting inside 16K

Treat the window as a budget with four line items, and reserve for the last one first:

• System turn. Fixed, and it is charged on every request. On a 16K window a 2,000-token system prompt is 12% of your budget spent before the user says anything.
• Retrieved or pasted context. The variable part, and the part to truncate. Chunk and rank rather than sending everything.
• Conversation history. Grows without bound unless you cut it. On a 16K model a naive chat loop hits the ceiling within a session.
• Reserved output. Subtract this before you fill anything else. If you want 1,500 tokens of answer, your real input budget is about 14,500, not 16,000.

Count the input rather than estimating it. Phi-4 uses a roughly 100K-entry vocabulary, so a rule of thumb derived from a 32K-vocabulary model will underestimate how much text fits — see where the Phi-3 tokenizer came from for why the two families count differently.

The history line is the one that catches production systems, because it is the only item that grows on its own. A chat loop that appends every turn hits 16K after enough exchanges no matter how short each one is, and the failure arrives mid-conversation for a user who has done nothing unusual. Pick a strategy before that happens: keep the last N turns, summarise older ones into a running note, or drop middle turns while preserving the first exchange, which usually carries the task definition. All three are worse than infinite context; none of them is worse than a request that fails at turn twelve.

It is also worth reserving output before you fill the prompt rather than after. The natural way to write this code — assemble as much context as fits, then generate — silently leaves nothing for the answer on exactly the requests that had the most to say. Subtract the reservation first and truncate context against the remainder.

The rest of the Phi-4 line

“Phi-4” is now a family rather than a checkpoint, and the context length is one of the things that varies across it. The 14B dense model, the smaller mini line, the multimodal variant and the reasoning-tuned releases each carry their own figure on their own card, and they are not the same. The mini and multimodal variants in particular are documented at substantially longer windows than the 14B flagship, which inverts the usual expectation that the bigger model in a family has the bigger window.

Because of that, the only safe habit is the one at the top of this page: pull max_position_embeddings off the exact repository you are loading, and pin the revision so it does not change under you. Pinning a Phi checkpoint covers how.