Why the First Token Takes Longer Right After a Model Loads

The first request after a model loads is slower than the tenth, often by several seconds, and it is not because prefill is slower the first time. Four separate one-time costs are being paid, they have different magnitudes, and three of them can be sized from numbers the hardware vendors publish. This page does that arithmetic. For what those phases are doing, read loading versus warming up alongside it.

Four costs, only one of which is prefill

• Reading weights from storage into page cache. Paid once per boot, or once per file eviction. Bounded by your disk.
• Copying weights across PCIe into VRAM. Paid once per load. Bounded by the link width and generation.
• Creating the driver context and loading kernels. Paid once per process. Bounded by nothing you control.
• Allocating the KV cache and compute buffers, and running the first forward pass. Paid once per model instance, on the first token that actually goes through.

Prefill itself is not on that list. Prefill for a given prompt costs the same on request one as on request one hundred; it is simply arriving after the other four rather than instead of them.

Reading the weights: bounded by your disk

A Q4_K_M quantization of an 8B model is about 4.58 GiB, per the llama.cpp quantize README’s size table, which is roughly 4.92 GB in decimal units. Divide that by the sequential read rate your drive is rated for and you have the floor for a cold read:

4.92 GB / 7.0 GB/s  (PCIe 4.0 NVMe, typical rated sequential)  = 0.70 s
4.92 GB / 3.5 GB/s  (PCIe 3.0 NVMe, typical rated sequential)  = 1.41 s
4.92 GB / 0.55 GB/s (SATA III SSD, interface-limited)          = 8.95 s
4.92 GB / 0.15 GB/s (5400rpm spinning disk, optimistic)        = 32.8 s

Those rates are the manufacturers’ own sequential figures for their respective classes, and they are ceilings; a real read gets less. The important consequence is the second load. Because llama.cpp maps the file rather than reading it, the pages stay in the operating system’s page cache after the first load, so the second load of the same file on a machine with enough free RAM costs approximately nothing. That is why “it was slow the first time and fast after I restarted it” is the single most common confusing observation here, and it is the page cache, not the GPU.

Getting them onto the card: bounded by the link

PCI-SIG’s published rate for PCIe 4.0 is 16 GT/s per lane with 128b/130b encoding, which gives about 1.97 GB/s per lane and about 31.5 GB/s across an x16 slot. Real host-to-device copies land below that; treating 25 GB/s as an achievable figure for a pinned-memory transfer:

4.92 GB / 25 GB/s  = 0.20 s   (PCIe 4.0 x16)
4.92 GB / 12 GB/s  = 0.41 s   (PCIe 3.0 x16, or 4.0 x8)
4.92 GB / 3 GB/s   = 1.64 s   (PCIe 3.0 x4 — a riser, or an M.2 adapter)

For a single 8B model this is the smallest of the four costs and not worth optimising. It stops being negligible when the file is 40 GB rather than 5, and it becomes the dominant cost on a machine where the card is on a reduced-width link — a mining riser, an eGPU enclosure, or a second slot that drops to x4 when both are populated. If load feels wrong on a large model, check the negotiated width before anything else with nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv.

The driver context and the first kernel launch

Creating a CUDA context is a per-process cost that has nothing to do with your model. The driver has to be resident, the context has to be created, the compiled kernels for your architecture have to be loaded, and library handles such as cuBLAS have to be initialised. On Linux without persistence mode, the driver itself may be unloaded when no process is using the GPU, and the first process to touch it pays for bringing it back — which is what nvidia-smi -pm 1 exists to prevent.

Nobody publishes a figure for this, and any number you see quoted for it is a measurement on somebody’s specific driver version and card. What you can say structurally is that it is per-process and not per-request, so a long-lived server pays it once at startup and a command-line invocation pays it every time. That is most of the reason a persistent server feels so much better than repeated one-shot runs, and it is the argument for running the thing under a service manager rather than launching it per query.

The fourth cost, the first forward pass, is where the KV cache is actually allocated and where any mapped weight page that has not been touched yet gets faulted in. llama.cpp runs a warmup pass at startup precisely to move that cost off the first real request; --no-warmup disables it, which makes startup look faster and the first request slower without changing the total.

Separating them on your own machine

The four costs are individually observable, and the method is subtraction. Load the model twice in a row and the difference is the disk read, because the second load hits page cache. Compare a request after a warmup pass against one with --no-warmup and the difference is the first-forward-pass cost. Compare a fresh process against a second request to a running server and the difference includes the driver context.

# 1. Cold: drop the page cache first (Linux, needs root)
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
/usr/bin/time -v llama-server -m model.gguf -c 4096 --port 8080 &
# llama.cpp prints its own "load time" line to stderr — read it

# 2. Warm: kill it and start it again without dropping caches.
#    The delta between the two "load time" lines is your disk read.

# 3. First-token cost, isolated to one token:
curl -s -o /dev/null -w '%{time_total}\n' localhost:8080/completion \
  -d '{"prompt":"hi","n_predict":1}'
# run it three times; the first is the outlier, the rest are the floor

Two things to hold on to when you interpret the result. First, none of this scales with prompt length, so if the gap between first and second request grows when you lengthen the prompt, you are looking at prefill and not at cold start. Second, on a machine where the model file is larger than free RAM, the page cache never holds all of it and every load is a cold load — which is a memory problem wearing a latency problem’s clothes.

Three cases where it gets much worse

The arithmetic above describes a single card with the whole model on it. Three common configurations break one of its assumptions badly enough to change the answer by an order of magnitude.

• The model does not fit and layers spill to CPU. With -ngl set below the layer count, part of the file is copied to VRAM and part is left in host memory. The PCIe transfer gets smaller, which sounds like a win, and the steady-state generation rate collapses because every token now crosses the bus. The cold-start figure improves while the thing you actually care about gets much worse, which is a good reason not to optimise load time in isolation.
• Several cards, one model. Splitting layers across two GPUs means two PCIe transfers, and if the cards are on different root complexes or one of them negotiated a narrower link, the slower of the two sets the pace. Check both with the pcie.link.width.current query above rather than assuming they match — a second card in the lower slot dropping to x4 is extremely common on consumer boards. Multi-GPU inference covers the rest of that geometry.
• The runtime unloads between requests. Ollama releases a model after an idle timeout, so a machine serving one request an hour pays the entire cold-start cost on every single request, forever. This is the case that produces the complaint “it takes twenty seconds and then it is instant” from somebody whose page cache is fine and whose disk is fast: nothing is wrong with the load, it is simply happening again. Keeping the model resident trades idle power for that latency.

There is also a platform difference worth knowing about if you are comparing notes with somebody on another operating system. On Windows, the display driver model manages GPU memory itself and can page allocations out to host memory under pressure, so a model that has been resident and untouched while something else used the card can be partially in host memory when the next request arrives — a second, invisible cold start with no file read involved. On Linux the allocation stays where it was put until something frees it. The same hardware, the same model and the same flags can therefore produce genuinely different first-request behaviour, and neither machine is misconfigured.