Benchmarking Your Own Local Model's Tokens per Second

“Tokens per second” is three different numbers, and quoting the wrong one is how two people with identical hardware end up disagreeing. Both major local runtimes already count what you need.

Decide which number you want

Before running anything, pick which of these you are measuring, because they differ by an order of magnitude on the same machine:

• Prompt processing (pp), or prefill. Tokens of input consumed per second. Compute-bound, highly parallel, and usually the largest of the three. llama.cpp calls this pp.
• Text generation (tg), or decode. Output tokens produced per second after the first. Memory-bandwidth-bound, and the number people mean when they say a model runs at N tokens per second. llama.cpp calls this tg.
• End-to-end. Total tokens divided by wall-clock time including model load and prefill. Honest about what a user experiences, useless for comparing hardware, because a cold load dominates it.

Report the first two separately and always with the conditions: quantization level, context length, prompt length, batch size and whether layers were on the GPU. A tg figure without a quant level is not a measurement of anything.

The purpose-built tool

llama.cpp ships llama-bench, which exists for this and handles repetition and warmup for you. Its documented defaults are a 512-token prompt for pp, 128 generated tokens for tg, and 5 repetitions:

llama-bench -m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  -p 512 -n 128 -r 5

Output is a markdown table whose last column is tokens per second with a standard deviation across the repetitions, in the shape 132.19 ± 0.55. Report both halves. A large deviation relative to the mean is the run telling you something moved — thermal throttling, another process, a power profile change — and a single number hides exactly that.

The flags worth knowing: -t for thread count, -ngl for layers offloaded to GPU, -b and -ub for batch and micro-batch size, -r for repetitions, and -o for output format when you want to feed the results somewhere. All of them accept comma-separated lists, which is what makes the sweep useful:

# find the thread count that actually helps, rather than assuming all cores
llama-bench -m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  -p 512 -n 128 -t 4,6,8,12,16 -r 3

That sweep is the single most valuable thing on this page for a CPU-only machine, because thread count usually peaks well below the core count once the memory controller saturates — the reason is in the bandwidth ceiling, and the peak is different on every machine.

Measuring your actual server

llama-bench measures the runtime in isolation. If you want the number for your deployment — with your context size, your system prompt, your slot configuration — measure the server you are running. Both servers report their own counters, which is better than timing from outside because it separates prefill from decode for you.

llama.cpp’s server returns a timings object on non-streaming completions:

"timings": {
  "prompt_n": 12,
  "prompt_ms": 371.5,
  "prompt_per_second": 32.30,
  "predicted_n": 35,
  "predicted_ms": 661.06,
  "predicted_per_second": 52.94
}

predicted_per_second is your decode rate and prompt_per_second is your prefill rate, both computed by the server from its own clocks. For streaming responses, set timings_per_token to get the same information as generation proceeds.

Ollama returns nanosecond counters and its API documentation gives the formula explicitly — divide eval_count by eval_duration and multiply by 10^9:

curl -s http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Write four sentences about memory bandwidth.",
  "stream": false
}' | python -c "
import json,sys
d = json.load(sys.stdin)
print('prefill  %.1f tok/s' % (d['prompt_eval_count'] / d['prompt_eval_duration'] * 1e9))
print('decode   %.1f tok/s' % (d['eval_count'] / d['eval_duration'] * 1e9))
print('load     %.2f s'     % (d['load_duration'] / 1e9))
"

Note load_duration in that output. On a cold call it will be seconds and it is not part of either rate — which is exactly why computing tokens per second from your own stopwatch gives a much worse number than the model is capable of, and why the counters are the right source.

A repeatable script

1. Fix the conditions and write them down. One model file, one quant level, one context size, one prompt. Record the filename, the runtime version (llama-server --version), and whether the GPU was used. Without these the number cannot be compared with anything, including your own run next month.
2. Warm up and discard the first run. The first request pays model load and page faults. Every subsequent run measures the steady state, which is what you are after.
3. Run enough repetitions to see the spread. Five is the tool’s default and a reasonable floor. Report the median rather than the mean, because a single thermal excursion drags a mean past every request that actually happened.
4. Drive it from a script so the conditions cannot drift between runs:

   #!/usr/bin/env bash
   set -euo pipefail
   URL=http://127.0.0.1:8080/v1/chat/completions
   PROMPT="Explain memory bandwidth in exactly one paragraph."
   
   request() {
     curl -s "$URL" -H "Content-Type: application/json" -d "$(cat <<JSON
   {"model":"local","messages":[{"role":"user","content":"$PROMPT"}],
    "max_tokens":128,"temperature":0,"stream":false}
   JSON
   )"
   }
   
   request > /dev/null            # warmup, discarded
   for i in $(seq 1 5); do
     request | python -c "
   import json,sys
   t = json.load(sys.stdin)['timings']
   print('%.2f' % t['predicted_per_second'])
   "
   done | sort -n | awk '{a[NR]=$1} END {print "median tg:", a[int((NR+1)/2)], "tok/s over", NR, "runs"}'

5. Compare against the ceiling, not against somebody else. Divide your machine’s memory bandwidth by the model file size. If your measured decode rate is a reasonable fraction of that, the setup is working; if it is a small fraction, something is wrong — layers on the CPU that should be on the GPU, a single memory channel populated, a thread count past the useful point.

What makes the number move

Before you publish a figure or use it to decide anything, know which of these applied, because each one is worth a large fraction:

• Thermals. A laptop sustains a much lower clock after two minutes than in the first ten seconds. Run long enough to reach steady state, and report which one you measured.
• Another model resident. A second model holding VRAM or RAM changes what fits and can silently push layers onto the CPU. Ollama in particular keeps models loaded for five minutes after their last use by default.
• Page cache. The first load reads from disk; later ones may come from cache. This affects load time strongly and decode rate weakly, which is another reason to separate them.
• Context length and cache reuse. Decode slows as the KV cache grows, because there is more of it to read per token. A rate measured at 100 tokens of context is not the rate at 30,000.
• Batch and slot configuration. A server with several parallel slots divides its context among them and behaves differently under concurrent load than under one request — which is worth checking if you run more than one model.
• Warmup. llama.cpp performs a warmup run by default and --no-warmup disables it. Whichever you choose, choose the same one every time.

A tokens-per-second figure with the conditions attached is a measurement somebody else can reproduce. The same figure on its own is a rumour, and the local-model corner of the internet is full of them — which is the reason to generate your own rather than look one up.