Running a Small Chat Model on a Raspberry Pi

A Raspberry Pi 5 will hold a conversation with a 1B-class model. It will not do it quickly, and the reason is arithmetic you can do before you start: decode speed is capped by memory bandwidth divided by the size of the weights.

What the hardware can physically do

Raspberry Pi specifies the Pi 5 with LPDDR4X-4267 SDRAM, in capacities up to 16 GB (Raspberry Pi’s product page). Turn that into a bandwidth figure with one stated assumption — a 32-bit memory interface, which is what the BCM2712 uses:

4267 MT/s x 32 bits / 8 bits per byte = 17,068 MB/s
                                     ~ 17.1 GB/s theoretical peak

That is a peak, not an achievable rate; no DRAM subsystem reaches its theoretical figure, so the real ceiling is lower and the arithmetic below can only be optimistic. Now the weights. Generating one token requires reading every weight once, so:

tokens/second  <=  memory bandwidth / bytes of weights

For sizes, use published file sizes rather than a remembered bits-per-weight number. bartowski’s Llama-3.2-1B-Instruct-GGUF model card lists F16 at 2.48 GB and Q4_K_M at 0.81 GB — a ratio of 0.327, so Q4_K_M costs about 5.2 bits per weight once the K-quant scale metadata is counted. Dividing:

17.1 GB/s / 0.81 GB = ~21 tokens/second   (upper bound, 1B at Q4_K_M)
17.1 GB/s / 2.48 GB = ~7  tokens/second   (upper bound, same model at F16)

Those are ceilings derived from published specifications, not measurements. What they tell you is the shape of the problem: quantization roughly triples the ceiling, a 7B model at the same quantization would divide it by seven, and no amount of thread tuning moves the bound at all. The last section shows how to find where your board actually lands.

Practically, that means a sub-2B model. Anything larger is not “slow”, it is unusable for interactive chat, and the 8 GB and 16 GB Pis buy you context length rather than speed.

One more asymmetry to expect before you are surprised by it. Reading the prompt and writing the answer are different operations on different bottlenecks. Prefill processes every prompt token in parallel and is bounded by arithmetic, so it uses all four cores properly; decode produces one token per pass and is bounded by memory, so extra cores mostly wait. On a Pi that shows up as a long pause after you press enter, followed by output that then arrives at a steady rate no matter how long the prompt was. Those are two numbers to keep separate, and llama-bench reports them separately for exactly that reason.

Building llama.cpp

A 64-bit OS is required — check with uname -m, which must print aarch64. Then:

1. Install the build dependencies.

   sudo apt update
   sudo apt install -y build-essential cmake git libcurl4-openssl-dev

2. Clone and build. There is no GPU backend worth enabling here; the CPU backend with ARM NEON is the fast path.

   git clone https://github.com/ggml-org/llama.cpp
   cd llama.cpp
   cmake -B build -DCMAKE_BUILD_TYPE=Release
   cmake --build build --config Release -j4

3. Confirm the binaries. The tools land in build/bin/. You want llama-cli, llama-server and llama-bench. Older guides refer to a binary called main; that name was retired and its absence is not a build failure.

Choosing and fetching a model

Pick a sub-2B instruction-tuned model in GGUF at Q4_K_M. Llama 3.2 1B Instruct, Qwen2.5 1.5B Instruct and Gemma 3 1B are all in this class. Note that some of these families gate access to the original weights behind a licence acceptance on the publisher’s model page — accept it there and download from your account rather than looking for a way round it.

mkdir -p ~/models
cd ~/models
curl -L -o llama-3.2-1b-instruct-q4_k_m.gguf \
  "https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf"

Put that directory on a USB SSD rather than the boot SD card if you have one. A GGUF download is a multi-gigabyte sequential write and the model file is then read repeatedly — what that does to an SD card over time is the reason this is not a style preference.

Running it

1. One-shot, to confirm it loads.

   ~/llama.cpp/build/bin/llama-cli \
     -m ~/models/llama-3.2-1b-instruct-q4_k_m.gguf \
     -p "Explain what a memory bandwidth bound is, in two sentences." \
     -n 128 -t 4 -c 2048 -no-cnv

2. Set the thread count to the performance cores, not to everything. The Pi 5 has four Cortex-A76 cores, so -t 4. Going higher does not help: there is nothing else to run on, and oversubscribing adds scheduling overhead to a workload that is already waiting on memory.
3. Keep the context small. -c 2048 rather than the model’s maximum. Llama 3.2 1B’s published config gives 16 layers, 8 key-value heads and a head dimension of 64, so at float16 the KV cache costs 2 x 16 x 8 x 64 x 2 = 32,768 bytes per token — 32 KiB. At 2,048 tokens that is 67 MB. At the model’s full 131,072-token context it is 4.3 GB, several times the size of the weights, and on an 8 GB Pi it is the thing that fills the machine.
4. Run it as a server if you want to use it from anywhere else. llama-server exposes an OpenAI-compatible endpoint on port 8080:

   ~/llama.cpp/build/bin/llama-server \
     -m ~/models/llama-3.2-1b-instruct-q4_k_m.gguf \
     -c 2048 -t 4 --host 0.0.0.0 --port 8080

   That binds to every interface with no authentication, so keep it on a trusted network or put something in front of it.
5. Cool the board before you judge it. The Pi 5 throttles under sustained load, and a generation long enough to be interesting is long enough to trigger it. Watch it live with vcgencmd measure_temp and vcgencmd get_throttled — a non-zero result from the second means the numbers you are looking at are the numbers for a hot board, not for the hardware. An active cooler changes results here more than any runtime flag.

Measuring it properly

The number you want is not the one scrolling past in the chat. llama.cpp ships llama-bench for exactly this, and it separates the two rates that matter — pp for prompt processing, which is compute-bound and parallel, and tg for token generation, which is the bandwidth-bound number the derivation above predicts:

~/llama.cpp/build/bin/llama-bench \
  -m ~/models/llama-3.2-1b-instruct-q4_k_m.gguf \
  -p 512 -n 128 -t 4 -r 5

It prints a markdown table with a t/s column and a standard deviation, repeated -r times. Run it with the board at a stable temperature and with nothing else loaded; the Pi throttles, and a first run after boot is not comparable to the fifth. Compare the tg figure against the ~21 tokens/second ceiling derived above: the gap between them is your board’s real achieved bandwidth and the runtime’s overhead, and it is the only honest version of this number that exists for your hardware.

Sweep rather than measuring once. Passing several values to a flag makes llama-bench run the whole grid, which is how you find out whether a knob does anything on this machine instead of assuming it does:

llama-bench -m ./model.gguf -p 512 -n 128 -t 2,3,4 -r 3
llama-bench -m ./model-q4_k_m.gguf -m ./model-q8_0.gguf -p 512 -n 128 -t 4

What the derivation predicts for those two sweeps is worth writing down before you run them, so the result can contradict you. Generation should stop improving with thread count well before four, because threads add arithmetic capacity to a step that is waiting on memory. And a Q8_0 file is roughly twice the bytes of a Q4_K_M one, so its generation rate should fall close to proportionally while its prompt-processing rate should not. If your board disagrees with either, the bandwidth model is not describing it and the interesting work is finding out why — which is a better outcome than a number copied from this page.