Phi-3's Context Window: Mini, Small, Medium and the Long-Context Variants

Phi-3 does not have a context window. It has six published checkpoints with three different documented lengths, and the 128K versions are a different download rather than a flag you set. Here is the whole grid, and what the long one costs when you try to fill it.

Every checkpoint and its documented length

Microsoft publishes a model card for each Phi-3 and Phi-3.5 release on Hugging Face, and the context length is stated on each one. The repository name carries it: Phi-3-mini-4k-instruct and Phi-3-mini-128k-instruct are the same parameter count with different position handling.

checkpoint                        params        context
Phi-3-mini-4k-instruct            3.8B          4K
Phi-3-mini-128k-instruct          3.8B          128K
Phi-3-small-8k-instruct           7B            8K
Phi-3-small-128k-instruct         7B            128K
Phi-3-medium-4k-instruct          14B           4K
Phi-3-medium-128k-instruct        14B           128K
Phi-3.5-mini-instruct             3.8B          128K
Phi-3.5-MoE-instruct              16x3.8B       128K
                                  (6.6B active)
Phi-3.5-vision-instruct           4.2B          128K

Two things in that table are worth reading twice. The base lengths are not uniform — mini and medium start at 4K, small starts at 8K — so a system written against “Phi-3 has a 4K window” will silently truncate differently when somebody swaps the size. And the Phi-3.5 line has no short variant at all: every Phi-3.5 checkpoint is documented at 128K, which is Microsoft consolidating on the long configuration rather than shipping both.

The numbers are in tokens, not characters or words. Phi-3-mini uses a 32,064-entry vocabulary, which affects how much English, and especially how much non-English and code, fits into 4,096 of them — see where the Phi-3 tokenizer came from.

Why 4K and 128K are separate models

Context length is not a buffer size. It is a property of how the model encodes position, and Phi-3 uses rotary position embeddings. A model trained with RoPE at 4K has never seen the rotation angles that correspond to position 90,000; feed it a sequence that long and the positional signal is out of distribution, which shows up as fluent output that has stopped tracking what is actually in the prompt.

The 128K checkpoints are the short ones extended with LongRoPE, the method Microsoft published in 2024 as “LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens”. It searches for a non-uniform rescaling of the RoPE frequencies — different dimensions are stretched by different factors — and then fine-tunes at the longer length. The result is a set of per-dimension scaling factors baked into the checkpoint, which is why they appear in config.json as a rope_scaling block containing long and short factor arrays rather than a single number.

The practical consequence: you cannot turn a 4K Phi-3 into a 128K one by raising max_position_embeddings. The value in the config is a declaration of what the weights were tuned for, not a setting that grants the capability. If you need the long window, download the long repository.

What 128K costs in memory

The 128K number on the card is a capability, not a promise that you can afford to use it. Every token in the context has to keep its attention keys and values resident for the whole generation, and that cache is sized by the architecture, not by the parameter count.

Here is the arithmetic for Phi-3-mini, with every assumption named. Its config.json documents 32 hidden layers, 32 attention heads and a hidden size of 3,072, giving a head dimension of 3072 ÷ 32 = 96. It uses standard multi-head attention, so there are 32 key heads and 32 value heads — not the smaller grouped-query count some models of this size use. Assume a 16-bit KV cache at 2 bytes per element and no cache quantisation:

bytes per token = 2 (K and V)
                x 32 layers
                x 32 kv heads
                x 96 head dim
                x 2 bytes  (fp16/bf16)
                = 393,216 bytes  = 384 KiB per token

at   4,096 tokens : 384 KiB x   4,096 =  1.5 GiB
at  32,768 tokens : 384 KiB x  32,768 = 12.0 GiB
at 131,072 tokens : 384 KiB x 131,072 = 48.0 GiB

The weights of a 3.8B model in bf16 are about 7.6 GB. The cache at full context is roughly six times that. This is the number that decides whether the long variant is usable on your hardware, and it is not on the model card — the card gives you the inputs, and the multiplication is yours to do.

Documented length is not usable length

Three separate ceilings sit underneath the number on the card, and you hit whichever is lowest:

• The documented length — what the weights were tuned for. Exceeding it is a quality failure, sometimes a silent one.
• The memory ceiling — the KV arithmetic above. A serving framework will pre-allocate for the maximum sequence length you configure, so declaring 128K on a 24 GB card fails at startup rather than at request time.
• The attention-cost ceiling — prefill work grows with the square of sequence length. A 100K-token prompt to a 3.8B model can take longer to read than a large model takes to read a short one, which undoes the reason you picked a small model.

Long-context capability in a small model is best treated as headroom for occasional large inputs rather than as a design centre. If most requests are long, the small end stops being cheap.

There is a fourth ceiling that no configuration file reports, and it is the one to be most sceptical about: how much of a long context the model can actually use. A window extended by rescaling position embeddings and a short fine-tune is a window the model can attend across, not one it attends across evenly. Retrieval accuracy from the middle of a very long prompt is generally worse than from either end, and that gap is wider for a 3.8B model than for a frontier one, because the same rescaling has to be absorbed by far less capacity. Microsoft’s cards publish long-context evaluations for the extended variants; read them rather than assuming the nominal number is the operating number, and test recall on your own documents before designing around it.

Choosing between the short and long variant

Because they are separate downloads, this is a decision you make once per deployment rather than per request. Three things decide it:

• The 95th percentile of your input length, not the maximum. If almost every request fits in 4K and a handful do not, the cheaper arrangement is usually the short variant plus an explicit path for the outliers — truncate, summarise, or route them elsewhere — rather than paying the long variant’s memory footprint on every request.
• Whether you can afford the cache at your concurrency. The 48 GiB figure above is for one sequence at full length. Ten concurrent requests at 32K each is 120 GiB of cache, which is a different class of hardware from the one a 3.8B model suggests. Serving frameworks let you cap the maximum sequence length below the model’s declared one for exactly this reason, and doing so is usually the right call: declare 16K on a 128K checkpoint and you get the long weights with a footprint you can predict.
• Whether prefill latency is on your critical path. If a user is waiting, a 60K-token prompt to a small model is a long silence before the first token. Batch and offline workloads do not care; interactive ones care a great deal.

The default worth starting from is the short variant with a declared maximum you have actually budgeted for, moving to the long one when you have a measured input distribution that needs it. Reaching for 128K because it is available is how a small model ends up costing more to serve than the large one it was meant to replace.

How to confirm it for your copy

Read it out of the checkpoint you actually loaded rather than from memory of the card:

from transformers import AutoConfig

cfg = AutoConfig.from_pretrained("microsoft/Phi-3-mini-128k-instruct")
print(cfg.max_position_embeddings)   # declared context length
print(cfg.num_hidden_layers, cfg.num_attention_heads, cfg.hidden_size)
print(getattr(cfg, "rope_scaling", None))  # present on the 128K variants

If rope_scaling is None you are holding a short-context checkpoint, whatever the directory is named. That single check catches the most common version of this mistake, which is a cached download from before somebody switched repositories.