Llama 3.2's 1B and 3B Models: Context Window at the Small End

Meta gives the Llama 3.2 1B and 3B models a documented context length of 128K tokens — the same figure as the 405B. On a model this small, that window is not the constraint you expect it to be, and it is not the one your local install will give you.

The documented figure

Llama 3.2 was announced in September 2024 with four sizes: 1B and 3B text-only models, and 11B and 90B vision models. The model card in Meta’s llama-models repository states a context length of 128K for the 1B and 3B, and the checkpoints’ config.json carries "max_position_embeddings": 131072, the same binary-rounded value as the rest of the family. The documented knowledge cutoff is December 2023, inherited along with everything else — see the cutoff table.

The figure is easy to misread on a download page, because Llama 3.2 is two releases wearing one version number. The 1B and 3B are text-only models intended for edge deployment; the 11B and 90B are vision models with an entirely different architecture bolted to a Llama 3.1 language backbone. They were announced together and they share a version, a licence and a documented context length, and almost nothing else. If a tutorial about “Llama 3.2” assumes image input, it is not about the models on this page.

Where the window came from

The small models were not trained from scratch to 128K. Meta describes them as produced by pruning and knowledge distillation from the larger Llama 3.1 models: structured pruning to reach the parameter budget, then distillation using the 8B and 70B as teachers. The long-context capability came with the architecture they were derived from, including the same scaled rotary position encoding described in RoPE scaling.

This matters for expectations. A 1B model with a 128K window can address 128K tokens; it does not follow that it can reason across them. Capacity to attend and capacity to use what it attended to are different, and the second scales with parameters. Treat the window on these models as room for retrieved context and long documents to be summarised, not as room for a task that needs the whole span held together at once.

The cache is bigger than the model

Here is the fact that surprises people, and it is arithmetic rather than opinion. Key-value cache per token is:

bytes_per_token = 2 (K and V) x layers x kv_heads x head_dim x bytes_per_value

Substituting the published configuration for Llama 3.2 1B — 16 layers, 8 key-value heads, head dimension 64 — in bf16:

2 x 16 x 8 x 64 x 2 bytes   = 32,768 bytes = 32 KB per token

32 KB x 128,000 tokens      ≈ 4.2 GB of KV cache
1.24B params x 2 bytes      ≈ 2.5 GB of weights

The cache for one full-window request is larger than the entire model. For the 3B — 28 layers, 8 key-value heads, head dimension 128 — the gap widens:

2 x 28 x 8 x 128 x 2 bytes  = 114,688 bytes = 112 KB per token

112 KB x 128,000 tokens     ≈ 14.7 GB of KV cache
3.2B params x 2 bytes       ≈ 6.4 GB of weights

Assumptions, stated: bf16 weights and bf16 cache, no quantisation of either, a single sequence at full length, and no memory counted for activations or framework overhead. Read the three fields from your own checkpoint’s config.json — num_hidden_layers, num_key_value_heads and head_dim (or hidden_size / num_attention_heads) — and substitute rather than trusting the numbers above for a fine-tune or a quantised build.

The conclusion is what matters: on a small Llama, memory planning is about the cache, not the weights. Fitting a 1B model on an 8 GB device is easy; fitting a 1B model with a 128K window on an 8 GB device is not, and one long request can use more memory than four copies of the model.

Why your install gives you less

Because of the arithmetic above, every local runtime sets a default context well below the documented maximum, and that default — not the model card — is what you are running against. Ollama’s per-request num_ctx, llama.cpp’s --ctx-size and vLLM’s --max-model-len each cap it, and each has its own default that moves between versions.

On Ollama in particular, the default has historically been small enough that a long prompt is silently truncated rather than rejected, which is the failure described in exceeding a self-hosted context length. If you downloaded a 3.2 model to process long documents and it seems to ignore the first half of them, this is almost certainly why, and the fix is one option rather than a different model.

There is a second reason a local install gives you less, and it is easier to miss because it produces no message at all. A quantised build’s metadata carries its own context length, set by whoever converted it, and that value can be lower than the original checkpoint’s — sometimes deliberately, to make the file work on modest hardware out of the box. So the chain from Meta’s documented 128K to the number your process enforces has at least three links in it: the original config, the converter’s metadata, and the runtime flag. Only the last is yours, and only the last is something you can read from a command line. Check what the server logged at load time rather than reasoning from the first link.

The on-device case these sizes exist for

Meta positioned the 1B and 3B for edge and mobile deployment, and the memory arithmetic above is exactly why the context window is the first thing you have to decide on a device. A phone or a laptop has a fixed budget that the operating system will enforce by killing your process, and the cache is the part of the budget that grows with use.

Working the numbers the other way round is the useful move. Instead of asking “how much memory does 128K need”, ask “how much context does my budget buy”:

context_tokens = (memory_budget - weights) / bytes_per_token

# Llama 3.2 3B, 4-bit weights (~1.8 GB), 8-bit KV cache (56 KB/token),
# 3 GB budget:
(3.0 GB - 1.8 GB) / 56 KB  ≈  21,000 tokens

Assumptions again, and they are the ones that move the answer most: 4-bit quantised weights rather than bf16, an 8-bit rather than 16-bit KV cache, and nothing reserved for activations, the tokenizer or the rest of the application. Substitute your own quantisation and your own budget; the shape of the calculation is what transfers.

Two device-specific effects that the arithmetic does not show:

• Prefill dominates the felt latency. Reading a long prompt is one pass over all of it, and on a mobile accelerator that pass is slow. A 20,000-token context on a phone can mean seconds before the first token, which is a product decision as much as an engineering one.
• Cache growth is invisible until it is not. A chat that has been open for an hour holds a cache proportional to its length. On a server that is a capacity question; on a device it is the reason the app was terminated in the background.

Using the window you actually have

• Set the window explicitly and size it to the job. Choose 8K or 32K deliberately if that is what your prompts need; the memory you free is memory available for concurrency.
• Quantise the cache before shortening the window. Runtimes that support an FP8 or 8-bit KV cache roughly halve the per-token figure above, which is usually a better trade than halving the context for a small model.
• Budget for concurrency, not for one request. A server handling eight simultaneous 16K conversations needs the same cache as one 128K request. On-device deployments serve one user, so they can spend it all on length; a shared endpoint cannot.
• Verify quality at length before committing. The documented window says the model will accept the tokens. Whether a 1B model answers well with 100K tokens of context is a question about your task, and the only honest way to answer it is to run your own evaluation at the length you intend to use.