Local Inference on an Orange Pi 5, by the Numbers

The Orange Pi 5 is sold with a 6 TOPS NPU and up to 32 GB of RAM. Only one of those figures decides what you can run, and it is not the one on the box.

What is actually published

From Orange Pi’s own specification for the Orange Pi 5: a Rockchip RK3588S — four Cortex-A76 plus four Cortex-A55 — with an NPU quoted at up to 6 TOPS, and LPDDR4/LPDDR4X memory offered in 4 GB, 8 GB, 16 GB and 32 GB configurations.

What is not published, on that page or anywhere official, is the memory clock and interface width of each SKU, and therefore the memory bandwidth. That omission matters more than anything on the spec sheet, for reasons the last section covers. Everything below that can be derived from published numbers is derived; the one thing that cannot is named as such.

How much RAM the weights take

Start from published file sizes rather than from a bits-per-weight constant. bartowski’s Llama-3.2-1B-Instruct-GGUF card lists F16 at 2.48 GB and Q4_K_M at 0.81 GB. The ratio, 0.327, is unit-independent and tells you Q4_K_M costs about 5.2 bits per weight once K-quant scale metadata is counted, against 16 for F16. Applying that to other sizes:

weight bytes ~= parameters x 5.2 / 8      (Q4_K_M)

 1B  ->  1e9 x 5.2/8  = 0.65 GB
 3B  ->  3e9 x 5.2/8  = 1.95 GB
 7B  ->  7e9 x 5.2/8  = 4.55 GB
 8B  ->  8e9 x 5.2/8  = 5.20 GB
14B  -> 14e9 x 5.2/8  = 9.10 GB
32B  -> 32e9 x 5.2/8  = 20.8 GB

Now subtract the machine. A headless Debian or Ubuntu image plus your own service is on the order of 1–2 GB, and the Mali GPU and display stack take a further slice if you are running a desktop. Against the published capacities:

• 4 GB — 1B and 3B at Q4_K_M, with the 3B leaving very little for context. This is a classification and extraction board, not a chat board.
• 8 GB — 7B and 8B at Q4_K_M fit the weights with roughly 1–2 GB spare, which is a real but tight working budget once the KV cache below is counted.
• 16 GB — 14B at Q4_K_M comfortably, or an 8B with a long context.
• 32 GB — 32B at Q4_K_M fits on paper at 20.8 GB. Whether it is usable is a speed question, not a capacity question, and the answer is in the last two sections.

The context is a second budget

Weights are fixed. The KV cache grows with every token in the conversation, and it is the part people forget until the process is killed mid-generation. The formula is architectural, not empirical:

bytes per token = 2 (K and V)
                x layers
                x kv_heads
                x head_dim
                x bytes per element

Take Llama 3.2 1B, whose published config gives 16 layers, 8 key-value heads and a head dimension of 64. At float16:

2 x 16 x 8 x 64 x 2 = 32,768 bytes/token = 32 KiB/token

   4,096 tokens  ->  134 MB
  32,768 tokens  ->  1.07 GB
 131,072 tokens  ->  4.29 GB   (the model's full context)

A 1B model whose weights are 0.65 GB can therefore need 4.3 GB of KV cache at its advertised context length — six times the weights, and more than a 4 GB board has in total. Grouped-query attention is what keeps this survivable at all; a model with 32 key-value heads instead of 8 would cost four times as much per token. Set the context you need rather than the context the model advertises, and note that the same arithmetic scales up sharply for 7B and 14B models with more layers.

Why 6 TOPS is not the constraint

It is worth converting the NPU figure into the same units as everything else, because it explains why it does not appear in any of the budgets above. A decode step costs roughly two operations per active parameter. For a 3B model:

operations per token  ~= 2 x 3e9  = 6e9
NPU throughput         =  6e12 int8 ops/s
compute-bound ceiling  =  6e12 / 6e9  = ~1,000 tokens/second

Nothing on this board produces a thousand tokens per second, which tells you the arithmetic is not what you are waiting for. The same step must also read 1.95 GB of weights from DRAM, and DRAM on an SBC delivers single-digit to low-tens of GB/s. Decode is bandwidth-bound by a wide margin, and a TOPS figure describes the wrong axis.

There is a second reason the NPU does not appear: llama.cpp and Ollama do not use it. Reaching the NPU for text generation means converting the model with Rockchip’s separate RKLLM toolchain — see running inference on the RK3588’s NPU for what that involves and which model families it covers.

The number nobody publishes

Tokens per second for a given model on this board is not a published figure and cannot be derived, because the input the derivation needs — achieved memory bandwidth for your specific SKU and DRAM — is not published either. Anyone quoting one either measured it or made it up, and the two look identical in a blog post.

Measure both halves yourself. First the bandwidth, which is the term the ceiling depends on:

sudo apt install -y sysbench
sysbench memory --memory-block-size=1M --memory-total-size=16G \
  --memory-oper=read run

Then divide that figure by the weight size from the table above for a ceiling specific to your board, and measure the real rate with llama.cpp’s own benchmark harness, which separates prompt processing from generation:

llama-bench -m ./qwen2.5-3b-instruct-q4_k_m.gguf -p 512 -n 128 -t 4 -r 5

The tg row is the number you came for. Run it at a stable temperature with nothing else on the board, and treat any figure from somebody else’s board with a different RAM SKU as unrelated to yours.

Two things to control while you do it, because both will otherwise appear as differences in the model. The RK3588S has four Cortex-A76 and four Cortex-A55 cores, and the Linux scheduler will happily place inference threads on the small ones; pinning with taskset to the A76 cluster and setting the thread count to four is the comparable configuration. And this SoC throttles under sustained load like any other, so check the thermal zones between runs:

for z in /sys/class/thermal/thermal_zone*/temp; do echo "$z $(cat $z)"; done
taskset -c 4-7 llama-bench -m ./model.gguf -p 512 -n 128 -t 4 -r 5

The core numbering of the big cluster varies by kernel and device tree, so confirm it from lscpu -e rather than copying the range above.