Local LLM Tokens per Second on a Raspberry Pi 5, Derived

Nobody publishes tokens per second for a model on a Raspberry Pi, because it depends on the build, the quant and the thread count. What can be derived is the ceiling — and on this board the ceiling is low enough to make most of the decisions for you.

The one number that bounds everything

Generating one token requires reading every weight of the model once. There is no way around it at batch size one: the arithmetic per byte read is tiny, so the processor spends the step waiting on memory. That makes the token rate bounded above by memory bandwidth divided by the size of the weights, and it makes memory bandwidth the specification to look up rather than the clock speed or the core count.

Raspberry Pi publishes the Pi 5 as carrying LPDDR4X-4267 SDRAM on its product page (retrieved 11 August 2026), on a 32-bit memory interface. Those two facts give the number:

peak memory bandwidth = transfers_per_second * bus_bytes

  4,267 MT/s * (32 bits / 8) = 4,267e6 * 4 = 17.07 GB/s

Seventeen gigabytes per second, theoretical peak, shared between the four CPU cores, the GPU, the video engine and everything else on the board. For comparison, the desktop card on the 12GB page has 360 GB/s — twenty-one times more — and that ratio, not any difference in cleverness, is the whole story of why a Pi is slow at this.

The arithmetic

Divide the bandwidth by the weight bytes. File sizes are those published on the bartowski GGUF model cards for Llama-3.2-1B-Instruct and Llama-3.2-3B-Instruct (retrieved 11 August 2026):

ceiling (tokens/s) = 17.07e9 bytes/s / weight_bytes

  Llama 3.2 1B  Q4_K_M   0.81 GB  ->  21.1 tok/s
  Llama 3.2 1B  Q8_0     1.32 GB  ->  12.9 tok/s
  Llama 3.2 3B  Q4_K_M   2.02 GB  ->   8.5 tok/s
  Llama 3.2 3B  Q8_0     3.42 GB  ->   5.0 tok/s
  Llama 3.1 8B  Q4_K_M   4.92 GB  ->   3.5 tok/s

These are ceilings, derived, assuming perfect use of peak bandwidth and nothing else touching memory. They are not measurements and no measurement will exceed them. What they establish is the shape of the decision: a 1B model at Q4 is in the range where reading along with the output is tolerable; an 8B at Q4 is at walking pace and always will be, on this board, regardless of software.

The cache adds to this at long contexts. A 1B model has 16 layers, 8 KV heads and head dimension 64, so its fp16 cache is 2 × 16 × 8 × 64 × 2 = 32 KiB per token — a quarter of a gigabyte at 8,192 tokens, which the attention step must also read. Small at short contexts, not negligible at long ones.

Capacity is a separate question from rate and it bites first on the smaller boards. The Pi 5 ships in several memory sizes, and the model must fit in RAM alongside the operating system, because the alternative is fatal here in a way it is not on a desktop: if the weights spill to swap on a microSD card or a USB drive, every token re-reads them from storage at a rate three orders of magnitude below the memory bus. The system does not report an error; it simply stops making progress. On a 4 GB board, a 3B model at Q8 plus a desktop session is already over the line. Check with free -m before concluding a configuration is merely slow.

Why the real figure sits below the bound

• Peak bandwidth is not achieved bandwidth. A real memory controller achieves a fraction of theoretical peak, depending on access patterns and refresh. Anything above about 70% of peak on a single-channel LPDDR4X part is doing well.
• The memory is shared. The GPU, the video engine, USB and networking all contend for the same controller. A Pi running a desktop session and a model is dividing 17 GB/s between them.
• Quantized formats cost arithmetic to unpack. A k-quant block must be dequantized before it is used, and on four Arm cores that work is not free. This is the case where the compute side can genuinely become co-limiting rather than merely waiting.
• Thermal limits are real here. Sustained all-core load on a Pi 5 without active cooling will reach the throttle threshold, and a throttled clock reduces the rate at which the cores can issue memory requests.
• Prompt processing behaves differently. Prefill is compute-bound rather than bandwidth-bound, so it does not obey this ceiling at all — it is bounded by the cores. On a Pi, a long prompt can take longer than the answer.

Choosing a model for the bound you have

Work backwards from a rate you can live with. If you want at least 10 tokens per second, the ceiling arithmetic says the weights must be under 1.7 GB, which at roughly 4.9 bits per weight means a model of about 2.8 billion parameters or fewer. That is the entire selection criterion, and it is a considerably more useful one than a list of models somebody liked.

max parameters for a target rate, at ~4.9 bits/weight

  params <= (17.07e9 / target_tok_s) * 8 / 4.9

  target 20 tok/s -> 1.39 GB of weights -> ~2.3B parameters
  target 10 tok/s -> 1.71 GB            -> ~2.8B parameters
  target  5 tok/s -> 3.41 GB            -> ~5.6B parameters

Two caveats on using that. It ignores the losses in the previous section, so treat the parameter figure as an optimistic upper bound and aim below it. And a smaller model is not merely a faster one — the quality difference between a 1B and a 3B on anything requiring reasoning is large, so this arithmetic tells you what is possible rather than what is adequate.

Also check capacity, not just speed. A 16 GB Pi 5 has plenty of room for any model that meets the rate criterion; a 4 GB one does not, and on the smaller boards the storage the model is read from matters too — see storage wear with model files.

Measuring it yourself

The derivation gives a bound. Your board gives a number, and the tool for getting it is llama.cpp’s own benchmark, documented in the llama-bench README.

1. Build llama.cpp on the board itself rather than using a generic Arm binary, so the compiler targets this CPU’s instruction set.
2. Sweep the thread count, because more is not better past the core count and contention costs: llama-bench -m Llama-3.2-1B-Instruct-Q4_K_M.gguf -t 2,3,4 -n 128 -r 5.
3. Read the tg128 row for generation and the pp512 row for prompt processing separately. They are different bottlenecks and averaging them tells you nothing.
4. Divide your generation figure by the ceiling above. A result in the 50–70% range is what the hardware allows; far below that suggests thermal throttling or a build without the right optimisations, and vcgencmd measure_temp during the run will distinguish them.
5. Repeat at your real context length. The ceiling above is for weights only, and at 8k or more the cache read is a visible additional term.