Speculative Decoding in llama.cpp: Setting Up a Draft Model

Speculative decoding runs a small model ahead of a large one and lets the large one check its work in a single batched pass. The setup is four flags. Three of them were renamed, and the one most guides omit is the one without which nothing happens.

What the two models are doing

Generation is one token per forward pass, and a forward pass at batch size one is bounded by how fast the weights can be read from memory rather than by arithmetic. The hardware is idle in the arithmetic sense. Speculative decoding spends that idle arithmetic: a small draft model proposes several tokens cheaply, and the target model evaluates all of them in one pass, because evaluating n tokens at once costs barely more than evaluating one.

The target then compares. Every drafted token that matches what the target would itself have produced is accepted and kept; the first mismatch is replaced by the target’s own token and the rest of the draft is thrown away. The output distribution is the target model’s, not a blend — that is the property that makes the technique safe to switch on. What varies is only how many tokens you get per target pass.

The flags, after the rename

llama.cpp reorganised these under a --spec- prefix, and a guide written before that will hand you flags the binary now rejects. From the server reference in the llama.cpp repository:

• --spec-draft-model, aliased -md and --model-draft — the draft model GGUF. Default: unused.
• --spec-type — which speculation strategy to run. Default: none. This is the one people miss.
• --spec-draft-n-max N — how many tokens to draft per step. Default 3. This is the old --draft-max.
• --spec-draft-n-min N — below this many drafted tokens, the draft is discarded rather than verified. Default 0. Formerly --draft-min.
• --spec-draft-p-min P, aliased --draft-p-min — stop drafting once the draft model’s own confidence in its next token falls below this. Default 0.00.
• --spec-draft-ngl, aliased -ngld — GPU layers for the draft model, separately from the target’s -ngl. Default auto.

Because --spec-type defaults to none, a command that passes only -md loads a second model, spends memory on it, and drafts nothing. Name the type. For an ordinary small-model draft that type is draft-simple; the same flag also selects the model-free n-gram strategies and the EAGLE-3, MTP and diffusion draft heads, which are a different setup and not this page.

The draft model is a second model in every sense that costs you something. It has its own weights resident, its own KV cache, and its own placement: --spec-draft-device chooses which device holds it, and --spec-draft-type-k and --spec-draft-type-v (aliased -ctkd and -ctvd) set its cache types independently of the target’s. On a machine where the target already fills the GPU, the memory the draft needs comes out of the target’s layers or out of your context length, and that trade — fewer target layers offloaded, as covered in the -ngl page — is usually worse than no speculation at all.

Bringing up the pair

1. Build or install llama.cpp as normal — cmake -B build then cmake --build build --config Release. Nothing about speculation needs a special build.
2. Get two GGUF files from the same model family: a target you already serve, and a much smaller sibling that shares its tokenizer. The vocabulary has to match, and that constraint is strict enough to deserve its own page. Some family members are behind a gated repository; request access rather than looking for a mirror.
3. Start the server with both models and an explicit type:

   llama-server \
     -m  models/target-8b-Q4_K_M.gguf \
     -md models/draft-0.5b-Q4_K_M.gguf \
     --spec-type draft-simple \
     --spec-draft-n-max 5 \
     --spec-draft-p-min 0.75 \
     -ngl all -ngld all \
     -c 8192 --port 8080 --metrics

4. Send a request that generates a long, fairly predictable answer — speculation pays best on structured output, code and repetition, worst on terse creative prose:

   curl -s http://127.0.0.1:8080/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -d '{"model":"target","messages":[{"role":"user",
          "content":"Rewrite this JSON with camelCase keys, no commentary."}],
          "max_tokens":400}' | jq '.usage'

5. Read the acceptance counters, below. If you skip this step you have no idea whether you made anything faster.

Reading the acceptance rate

With --metrics enabled the server exposes a Prometheus endpoint carrying three counters that exist only for this feature: llamacpp:spec_decode_num_draft_tokens_total, llamacpp:spec_decode_num_accepted_tokens_total and llamacpp:spec_decode_num_drafts_total. All three read zero when speculative decoding is off, which is the fastest way to discover you forgot --spec-type.

curl -s http://127.0.0.1:8080/metrics | grep spec_decode

# acceptance rate = accepted_tokens_total / draft_tokens_total
# tokens gained per target pass = accepted_tokens_total / num_drafts_total

The second ratio is the one that decides whether this was worth doing. A draft of five tokens with two accepted on average means each target forward pass emits three tokens instead of one — the accepted two plus the target’s own correction. llama.cpp’s own speculative documentation reports the same statistic in the form draft acceptance rate = 0.57576 ( 171 accepted / 297 generated), so the shape of the number is the same whichever surface you read it from.

The /slots endpoint shows the per-slot settings actually in force as speculative.n_max, speculative.n_min and speculative.p_min, which is worth checking when a request overrides them.

When it makes things slower

Drafting is not free, and the arithmetic is unforgiving. Let the draft model cost a fraction f of a target forward pass, let n be --spec-draft-n-max, and let E be the average number of drafted tokens accepted. One speculative step costs about n·f + 1 target-pass-equivalents and yields E + 1 tokens, against one token for 1 without speculation. So the break-even condition is E > n·f.

Put numbers in it. A draft model a tenth the cost of the target, giving f = 0.1, with n = 5, needs more than 0.5 accepted tokens per step just to pay for itself. That is a low bar on JSON and a bar you can miss on free text. Raise n and the bar rises with it, linearly, while acceptance does not — each additional drafted token is conditional on all the previous ones being right, so the marginal token is the least likely to survive.

Two knobs push back on that. --spec-draft-p-min stops the draft early when the draft model itself is unsure, which cuts the wasted n·f on exactly the steps that were going to be rejected. --spec-draft-n-min throws away drafts that came out too short to be worth a verification pass. Both trade a little upside for less downside, and both are more useful than a large n-max.