Running Inference on the RK3588's NPU

The RK3588’s NPU is a fixed-function matrix engine with an offline compiler in front of it. Whether your model runs on it is decided entirely by that compiler, months before you power the board on, and the failure mode is not an error — it is silent fallback to the CPU.

What 6 TOPS actually buys

Rockchip specifies the RK3588 NPU at up to 6 TOPS, a figure repeated in board vendors’ own documentation — Firefly’s RK3588 NPU page gives “processing performance up to 6 TOPS”. That number is INT8 operations per second across the whole unit, and it is worth converting into something with units you care about before deciding what it means.

A single decode step of an autoregressive model costs roughly two arithmetic operations per active parameter — one multiply and one add. For a 3B model that is 6 x 10⁹ operations. Against 6 x 10¹² operations per second, the arithmetic alone would allow on the order of a thousand steps per second. Nobody gets anywhere near that, and the reason is not the NPU: it is that the same step has to read every one of those parameters out of DRAM once, which is a bandwidth problem the NPU does not solve. Treat 6 TOPS as evidence that compute is not your constraint, not as a throughput promise.

Where the NPU genuinely earns its place is convolutional vision work — detection, segmentation, classification — where the model is small enough to stay resident and the arithmetic per byte read is high. That is the workload the toolchain was built around, and it shows in everything below.

Nothing runs without the offline compiler

There is no path from a PyTorch checkpoint to the NPU at runtime. A model has to be converted to Rockchip’s .rknn container by rknn-toolkit2, which runs on an x86 host, not on the board. The board-side runtime, RKNPU2, loads that container and nothing else.

The practical route is PyTorch or TensorFlow to ONNX, then ONNX to RKNN. The conversion step is also where quantization happens, and it needs a calibration set — a few hundred representative inputs used to fix the per-tensor scales:

from rknn.api import RKNN

rknn = RKNN(verbose=True)
rknn.config(
    mean_values=[[0, 0, 0]],
    std_values=[[255, 255, 255]],
    target_platform="rk3588",
    quantized_dtype="asymmetric_quantized-8",
)
rknn.load_onnx(model="yolov8n.onnx")
rknn.build(do_quantization=True, dataset="./calib_list.txt")
rknn.export_rknn("./yolov8n.rknn")

Two things about that snippet decide most outcomes. target_platform bakes in the chip: an RKNN built for one Rockchip part will not load on another. And do_quantization=False is a real option — the NPU can execute float16 as well as int8 — but it doubles the bytes read per inference, which on a bandwidth-limited board is usually the wrong trade for a vision model and occasionally the right one when int8 calibration has destroyed accuracy.

The op set is the whole story

rknn-toolkit2 ships operator support tables in its repository, per toolkit version and per target chip, and they are the document to read before you plan anything. An operator that is not in the table for rk3588 does not fail the build. The compiler partitions the graph around it and the unsupported region executes on the Cortex-A76 cores instead.

That matters more than the raw count of supported ops, because of where the partition falls. An unsupported op in the middle of a backbone splits the graph in two and forces the intermediate tensor out to DRAM and back, which can cost more than the op saved. An unsupported op in a detection head — a non-maximum-suppression variant, an exotic activation — costs almost nothing, because it runs once on a small tensor. Two models with identical “95% of ops supported” can differ by an order of magnitude for this reason alone.

The other recurring constraint is shape. The compiler wants static shapes: input dimensions fixed at build time, no dynamic batch, no data-dependent control flow. Transformer graphs violate this constantly — attention with a growing sequence length is the canonical case — which is why vision transformers on this part are a research project and YOLO variants are a supported path. Quantizing at the ONNX layer before conversion does not remove this: shape rigidity is a property of the compiler, not of the numeric format.

Three cores, and one model uses one

The RK3588’s NPU is three cores, not one, and the runtime API makes that explicit. From rknn_api.h in Rockchip’s own repository, rknn_set_core_mask accepts RKNN_NPU_CORE_AUTO (documented as “run on NPU core randomly”), RKNN_NPU_CORE_0, _1 and _2, the combinations RKNN_NPU_CORE_0_1 and RKNN_NPU_CORE_0_1_2, and RKNN_NPU_CORE_ALL. The header notes the call is “only supported on multi-core NPU platform”.

The consequence is the thing people miss when a benchmark disappoints: the default is one context on one core. Three cores do not make one inference three times faster. They let you run three inference contexts concurrently — three camera streams, or three copies of the same model pinned to RKNN_NPU_CORE_0, _1 and _2 and fed from a thread pool. If your workload is a single stream at 30 fps, two thirds of the advertised 6 TOPS is idle by design, and the fix is a pipeline, not a flag.

Language models go through a different stack

rknn-toolkit2 is the vision path. Text generation has its own toolchain, rknn-llm, with its own converter (RKLLM-Toolkit), its own container format and its own C API. Its README lists RK3588, RK3576, RK3562 and RV1126B as target series and covers Llama, TinyLlama, Qwen 2/2.5/3, Phi, ChatGLM3, Gemma, InternLM2 and MiniCPM families, plus vision-language models such as Qwen2-VL and InternVL. Quantization appears in the model filenames as w8a8 and w4a16 — 8-bit weights with 8-bit activations, or 4-bit weights with 16-bit activations.

Do not assume llama.cpp or Ollama on this board are using the NPU. They are not: they run on the Cortex-A76 cluster, and a GGUF file has no route to RKNPU at all. Moving a language model onto the NPU means re-converting it with RKLLM-Toolkit and calling the RKLLM runtime. Several of the model families above ship under licences that gate access to the original weights; get them from the publisher on their terms and convert from there.

For what the same board can do without the NPU at all, and how much RAM each model size needs, see local inference on an Orange Pi 5, by the numbers.