Skip to content
Multigrid

Small model behaviour

Phi and Gemma at the small end: what a 1B–4B model's context window, tokenizer and multimodal input actually cost, and where the small end stops being a smaller version of the large one.

A 3B model is not a 70B model with the quality turned down. Most of what makes it different is not capability at all — it is that the small end is where you stop being a customer of an API and start being the operator of a runtime. Nobody hosts these for you by default. You choose the serving framework, the quantisation, the chat template and the stop conditions, and every one of those choices is a place the model can be made to behave incorrectly while still producing fluent text.

These pages are about that layer. What a documented context length costs in memory rather than what it says on the card, which tokenizer a model inherited and what that does to your token counts, how an image becomes a token bill, and why a model that was trained to stop keeps talking anyway. The figures come from Microsoft’s and Google’s own model cards and technical reports, cited in the sentence; where a figure is expected to move, the page says so rather than pretending otherwise.

Phi-3's Context Window: Mini, Small, Medium and the Long-Context Variants

The documented context length of every Phi-3 and Phi-3.5 checkpoint, why the 128K versions are separate downloads, and what 128K costs in KV cache.

9 min read

Phi-3's Chat Template and Special Tokens

The role tags Phi-3 was instruction-tuned on, annotated against a rendered conversation, and what breaks when you hand-roll them.

8 min read

Phi-4's Context Window and Output Limit

Phi-4's documented 16K window, why it is shorter than Phi-3's long variants, and why an open-weight model has no output limit of its own.

8 min read

Function Calling Support in Phi: Which Versions Added It

Which Phi releases document native tool-call tokens, which require a prompt-based workaround, and why the difference is a trained output channel rather than a feature flag.

9 min read

Phi-3's License: What the MIT Terms Permit

Phi-3 ships under MIT, so the licence imposes no user threshold, no acceptable-use policy and no naming requirement — and this page is precise about what it does still require.

8 min read

Phi-3's Tokenizer: Where It Was Reused From

Phi-3-mini and medium inherit the Llama-2 tokenizer padded to 32,064 entries, Phi-3-small uses a 100K one, and the split changes how much text fits in the window.

8 min read

Phi-3.5-Vision's Multimodal Input and Image Token Cost

How Phi-3.5-vision crops an image into 336-pixel blocks, the documented token-count formula, and what three real image sizes cost when you run it.

9 min read

The Documented Safety Training Behind Phi-3's Refusals

Microsoft's published break-fix methodology for Phi-3, and why a refusal from an MIT-licensed local model is a learned output rather than a filter.

8 min read

Phi Model Deprecation and Version History on Azure AI

How Azure AI Foundry retires models, where the authoritative Phi retirement dates live, and why an open-weight model's retirement removes an endpoint rather than the model.

8 min read

Phi-4's Output Determinism at Temperature 0

Why greedy decoding removes sampling randomness but not output variation, and which parts of a Phi-4 deployment have to be frozen before runs repeat.

9 min read

Pinning a Phi Checkpoint on Hugging Face

Load a Phi model at a specific Hugging Face commit instead of tracking main, verify the pin took effect, and mirror the checkpoint so it survives upstream changes.

9 min read

Why a Local Phi-3 Server Sometimes Misses the Stop Token

Phi-3 ends turns with <|end|> but its tokenizer's eos_token is <|endoftext|>; servers that only know the second run past the answer. Here is the fix per runtime.

9 min read

Gemma 2's Context Window, and Why It Is Shorter Than Its Peers

Gemma 2 ships an 8,192-token context at every size, and the interleaved-attention design that made the model cheap is the reason it stops there.

8 min read

Gemma's Chat Template: the start_of_turn Format

The exact turn-delimiter format Gemma instruction-tuned models were trained on, rendered in full and annotated line by line.

8 min read

Gemma's Tokenizer: a 256K Vocabulary Shared With Gemini

Gemma uses a SentencePiece vocabulary derived from Gemini's, unusually large for a small model, and that choice shows up in your token counts.

8 min read

Gemma's Licence: What Google's Terms of Use Actually Restrict

Gemma is not open source: it ships under a custom Google licence with a use policy attached that you must pass on to everyone you redistribute to.

9 min read

Gemma 2's Sliding Window Attention and Its Effect on Long Context

Gemma 2 alternates local 4,096-token attention with full-sequence attention layer by layer, which halves KV-cache growth and bounds how far the model can really reach.

9 min read

Gemma Has No System Role: Fixing the Template Error

Why apply_chat_template raises System role not supported on Gemma, and the fold-into-first-user-turn workaround Google's own template uses.

8 min read

Gemma 3's Image Input: Resolution, Tokens and Pan-and-Scan

Gemma 3's 4B and larger sizes accept images through a SigLIP encoder at a fixed square resolution, and non-square inputs are handled by cropping rather than by stretching.

8 min read

What Function Calling Gemma Supports Natively

No Gemma release ships tool-call tokens in its chat template, so function calling is a prompted convention you define and parse yourself.

8 min read

Gemma's Output Token Limit and Default Generation Length

Open-weight Gemma has no separate output ceiling: the bound is the context window minus your prompt, and the surprising defaults come from your runtime, not the model.

8 min read

Gemma's Version History: 1, 2 and 3

What each Gemma generation shipped, when, and which architectural change in it breaks assumptions carried over from the previous one.

9 min read

Pinning a Gemma Checkpoint Instead of Tracking Main

Load Gemma at a specific Hugging Face commit so a repository update cannot change your model, your tokenizer or your chat template overnight.

9 min read

Why Some Inference Servers Double-Emit Gemma's Stop Token

Gemma stops on end_of_turn rather than eos, and a serving stack that knows only one of them either runs past the end of the turn or leaves the marker in your text.

9 min read

Other topics