llama.cpp From Source to First Token

llama.cpp is a C++ inference engine with no Python dependency, its own model format, and the ability to split a model between GPU and CPU memory. That last property is why it runs models on hardware that cannot hold them, and why the performance question is always about memory bandwidth rather than about compute.

What llama.cpp is

Three design decisions define it. Models are stored in GGUF, a single self-contained file holding weights, metadata and the tokenizer. Quantisation is a first-class operation with many levels rather than a bolt-on. And layers can be placed on GPU or CPU individually, so a model that does not fit in video memory still runs — at the speed of whichever memory holds the slowest part.

It is also the engine underneath a large amount of local-AI software, including Ollama, which is worth knowing because performance characteristics you learn here transfer to those tools directly.

Building it

The project moved to a CMake build and renamed its binaries with a consistent prefix some time ago. Older instructions referring to a plain make and a binary called main describe a layout that no longer exists, which is the most common reason a copied command fails.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# CPU only
cmake -B build
cmake --build build --config Release -j

# With NVIDIA GPU support (requires the CUDA toolkit)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

# Binaries land in build/bin: llama-cli, llama-server, llama-quantize,
# llama-bench and others. Run any of them with --help; that output is the
# authoritative flag list for your commit.

The backend flag differs by platform — CUDA for NVIDIA, Metal for Apple silicon (on by default there), and others for ROCm and Vulkan. Build with the wrong one and everything works at CPU speed while appearing to succeed, which is worth checking before concluding your GPU is slow: the startup log states which backend was compiled in and how many layers were offloaded.

Getting a model into GGUF

Two paths. Most popular models already have GGUF conversions published on Hugging Face, and downloading one is a file transfer. If yours does not — a fine-tune of your own, most commonly — the repository includes a conversion script that reads a Hugging Face checkpoint directory and writes a GGUF.

# Convert an existing checkpoint directory to GGUF at full precision.
# The script name has changed across the project's history; check the
# repository root for the current one before running.
python convert_hf_to_gguf.py ./my-model-dir --outfile my-model-f16.gguf --outtype f16

Convert first, quantise second. The conversion output is a large file at the original precision; quantisation is a separate step that reads it. Skipping straight to a quantised conversion works for some paths and not others, and keeping the full-precision GGUF means you can produce a different quantisation later without re-converting.

Quantising, and what each level costs

./build/bin/llama-quantize my-model-f16.gguf my-model-q4_k_m.gguf Q4_K_M

# Run llama-quantize with no arguments to list the levels your build
# supports; the catalogue has grown and changed over time.

The naming carries real information once you know the pattern. The number is bits per weight. K denotes the k-quant family, which allocates different precision to different parts of the model rather than uniformly. The trailing letter is a size within that family — S, M and L for small, medium and large.

The size arithmetic is the useful part, and it is exact enough to plan with:

file_size ≈ parameters × bits_per_weight / 8

  7B at 16 bits  = 7e9 × 2      = 14.0 GB
  7B at  8 bits  = 7e9 × 1      =  7.0 GB
  7B at ~4.8 bits (a 4-bit k-quant, which averages above 4)
                 ≈ 7e9 × 0.6    ≈  4.2 GB

Add roughly 0.5–1 GB of runtime overhead, plus the KV cache:
  KV bytes/token = 2 × layers × kv_heads × head_dim × bytes_per_element

The practical rule that follows from bandwidth rather than from quality: a larger model at 4 bits usually beats a smaller model at 8 bits when both fit, because parameter count buys more than precision does over this range. The quality trade-offs of each level are covered in choosing a quantisation; what matters here is that the file size is predictable before you download anything.

Running it, and the server

# Interactive, with 35 layers offloaded to the GPU and 8k context
./build/bin/llama-cli -m my-model-q4_k_m.gguf -ngl 35 -c 8192 -p "Explain paging."

# As an OpenAI-compatible HTTP server
./build/bin/llama-server -m my-model-q4_k_m.gguf -ngl 35 -c 8192 --port 8080

Three flags carry most of the behaviour. -ngl is the number of layers placed on the GPU: raise it until video memory is nearly full, because every layer left on the CPU is served at system memory bandwidth, which is several times slower. -c is the context size, and it directly sizes the KV cache — asking for a large context on a nearly full GPU is the usual cause of an allocation failure at startup. The server exposes an OpenAI-compatible endpoint, so the same client code that talks to a hosted provider talks to it.

The ceiling: bandwidth over model size

This is the derivation that makes local performance predictable, and it needs no benchmark. Generating one token requires reading every active weight from memory once. So the maximum possible generation rate is memory bandwidth divided by the bytes that must be read per token.

tokens_per_second_ceiling ≈ memory_bandwidth_bytes_per_sec / model_bytes

Worked, with the two numbers you supply:

  a 4 GB quantised model on hardware with 200 GB/s of bandwidth
    200e9 / 4e9  =  50 tokens/s ceiling

  the same model on a laptop with 50 GB/s
    50e9 / 4e9   ≈  12.5 tokens/s ceiling

  a 40 GB model on the same 200 GB/s hardware
    200e9 / 40e9 =   5 tokens/s ceiling

Real output lands meaningfully below the ceiling — attention over the
KV cache, sampling and framework overhead all cost time — but the
ceiling is what changes when you change quantisation or model size,
and it is why halving the file size roughly doubles the rate.

Three things follow. Look up your hardware’s memory bandwidth, not its compute figure, when predicting generation speed. A mixed CPU-GPU split runs at close to the speed of the slower memory for the layers that live there, which is why offloading the last few layers has an outsized effect. And prompt processing behaves differently — it is compute-bound rather than bandwidth-bound, which is why a long prompt can be fast to read and slow to answer, the asymmetry described in memory bandwidth in inference.

Measuring your own machine

No figures are quoted on this page for a specific machine because none were taken. You do not need anyone else’s: llama.cpp reports its own timings on every run, and ships a benchmarking binary.

1. Read the timing block after any run. The tool prints, at the end of generation, the prompt evaluation time and rate and the generation time and rate as separate numbers. Those two rates are the two halves of the asymmetry above and should be reported separately, never averaged.
2. Run the bench binary for repeatable numbers. llama-bench takes a model and produces prompt-processing and token-generation rates without the noise of an interactive session. Run it with no arguments first to see the options your build has.
3. Sweep the offload count. Run at several values of -ngl and plot the generation rate. The curve is close to flat until the model fits, then rises sharply — the shape tells you exactly how much video memory a better result would need.
4. Compare against the ceiling. Divide your measured rate by the derived ceiling. Well above 50% means you are close to bandwidth-bound and further tuning will not help; far below means something is on the wrong device.
5. Record the model file, the quantisation, the flags and the hardware with every number. A token rate without those four is not a measurement, and it is why most quoted figures on the internet are useless to you.

The four failures you will hit

An unknown architecture on load. The engine reads the architecture name from the GGUF and needs code for it. A model released after your checkout was built will fail here, and the fix is to pull and rebuild rather than to hunt for a flag. The same applies in reverse: a GGUF produced by a much newer converter can use quant types an older binary does not implement.

Allocation failure with a large context. The KV cache is sized by -c at startup, so a value that was fine yesterday fails today because you offloaded more layers. Lower the context, or lower the offload count, and prefer setting the context to what your prompts actually need — the same reasoning as the vLLM arithmetic on serving with vLLM. Some builds also support a reduced-precision KV cache, which trades a little quality for a much longer usable context.

Everything running at CPU speed. Two causes, and the startup log distinguishes them: the binary was compiled without the GPU backend, or it was compiled with it and you did not pass a layer offload count. The log states the backend and reports how many layers were offloaded; read those two lines before drawing any conclusion about performance.

Output that never stops, or stops immediately. Both are prompt-format problems rather than model problems. The command line tools can apply the model’s chat template, and the server does so on its chat endpoint; bypassing that and concatenating strings yourself reproduces the failure described on the transformers page, where the model continues your text instead of answering it.