Tuning llama.cpp's Threads Flag for CPU-Only Inference

The advice is always “set threads to your physical core count”. That is a decent starting guess and it is frequently wrong, because the number that maximises generation and the number that maximises prompt processing are not the same number and llama.cpp gives you a separate flag for each.

Why the curve turns over

Generating one token means reading every weight the token needs, exactly once, and doing a small amount of arithmetic on each. The arithmetic-to-bytes ratio is terrible — roughly two operations per weight read — so on CPU the run is bound by how fast memory can feed the cores, not by how many cores there are.

Adding threads raises achieved memory bandwidth up to a point, because more outstanding requests keep more memory channels busy. Past that point the channels are saturated — the ceiling described in memory bandwidth and inference — and every additional thread only adds cost: synchronisation at each of the model’s many barriers, cache lines bouncing between cores, and on a hybrid CPU, work landing on efficiency cores that finish late and hold everyone up. The throughput curve therefore rises, flattens and then declines.

llama.cpp’s own llama-bench README shows this in its threads example. On a 7B Q4_0 on CPU, generation climbs 4.05 → 7.80 → 12.22 → 16.71 t/s at 1, 2, 4 and 8 threads, then falls to 15.32 at 16 and only recovers to 16.41 at 32. Over the same sweep prompt processing keeps climbing: 6.17 → 12.31 → 23.18 → 32.29 → 33.52 → 59.00. Those are that machine’s numbers on that model, and the useful part is the divergence: the two phases peak in different places, and one flag cannot serve both.

Two thread counts, because two phases

• -t/--threads N — threads during generation. This is the one that peaks early. Passing a value of 0 or less makes llama.cpp use hardware_concurrency, which counts hyperthreads.
• -tb/--threads-batch N — threads during batch and prompt processing. Documented as defaulting to the same as --threads, and this is the flag most people never set. Prompt processing is a matrix-times-matrix problem with real arithmetic intensity, so it keeps scaling past the point where generation has stopped.

The practical consequence: on a machine with more logical CPUs than memory bandwidth can feed, the right configuration is often -t at roughly the physical core count and -tb higher. Setting one number for both means either leaving prefill throughput on the table or dragging generation down.

Hyperthreads are the other common trap. Two logical cores on one physical core share the execution units and the L1/L2 cache. For a bandwidth-bound kernel there is little to interleave, so counting them usually adds contention without adding throughput. Start from physical cores and let the measurement tell you whether the extra logical ones earn anything.

Sweeping it on your own machine

Nobody publishes tokens per second for your CPU, your model and your quantization together, and no rule of thumb substitutes for twenty minutes of measurement. llama-bench accepts lists and ranges on every test parameter and reports a mean with a standard deviation over -r repetitions, which is what you need to tell a real difference from noise.

1. Find your physical core count. On Linux, lscpu reports sockets, cores per socket and threads per core; on macOS, sysctl -n hw.physicalcpu. Note it — you are testing around it, not obeying it.
2. Close everything else. A browser is enough to move these numbers by more than the effect you are looking for.
3. Sweep generation and prompt processing together, so you see the divergence:

   llama-bench -m model.gguf -ngl 0 \
     -p 512 -n 128 \
     -t 2,4,6,8,10,12,16 \
     -r 5

   -ngl 0 forces CPU-only, which is the point of the exercise; without it a GPU build will offload and measure something else. Leave -b and -ub at their defaults for this sweep — those are a separate axis and varying two at once tells you nothing about either.
4. Take the highest tg row as your -t and the highest pp row as your -tb. If two rows are within one standard deviation of each other, prefer the lower thread count — it leaves the machine usable and it is more stable under any other load.
5. Re-run the winner at the context you actually use, with -d to prefill the cache to a realistic depth. Thread scaling at depth 0 and at depth 8000 are not the same measurement.

Reading the result you get

A few shapes recur and each says something specific:

• Generation flat from 4 threads upward. You are bandwidth-saturated. More cores will never help; a smaller quantization or a GPU will, because both reduce bytes read per token.
• Generation peaks below your physical core count. Common on hybrid CPUs with efficiency cores, and on machines with fewer memory channels than cores. Trust it.
• Prompt processing still climbing at the top of your sweep. Extend the sweep for -tb only. Prefill has arithmetic to spare.
• Large standard deviations. Something else is running, thermal throttling has started, or the model is being re-read from disk. Check the load-mode flags before you trust any of the numbers.

One measurement caveat that llama.cpp states explicitly: llama-bench numbers exclude tokenization and sampling time. Your end-to-end rate in an application will be slightly lower, and the gap grows if you have an expensive sampler chain or a grammar attached.

Placement flags, for when the count is not enough

Where threads run can matter as much as how many there are, and llama.cpp exposes the controls rather than guessing:

• -C/--cpu-mask takes a hex affinity mask, with --cpu-strict to enforce it. On a hybrid CPU this is how you keep the model on performance cores instead of hoping the scheduler does.
• --poll (0–100) sets how aggressively worker threads spin rather than sleeping between barriers. High polling reduces wake-up latency and burns CPU; on a shared machine it is the wrong trade.
• --prio raises process and thread priority, which mainly helps by preventing preemption mid-barrier.
• --numa distribute|isolate|numactl matters on multi-socket servers, where a thread reading weights from the other socket’s memory pays a large penalty. isolate confines execution to the node it started on, which is usually the right answer for a single model that fits in one node’s memory.

Change one at a time and re-measure. These interact — a mask that restricts threads to six cores makes a -t 12 setting actively harmful — and a stack of changes applied together tells you nothing about which one did the work.