AI Accelerator HATs for the Raspberry Pi

An AI HAT does not make a Raspberry Pi faster at everything. It executes one compiled neural network graph and nothing else, and on most of these boards that graph cannot be a language model.

What is actually offloaded

These accelerators are not co-processors in the sense a GPU is. They take a neural network that has been compiled ahead of time into the vendor’s own binary format, hold its weights, and execute it on tensors you hand them. Everything outside that graph — the decisions, the glue, the I/O — stays where it was.

On the Raspberry Pi accessory line the accelerator is a Hailo part. Raspberry Pi’s AI HAT+ documentation lists the 13 TOPS variant as a Hailo-8L and the 26 TOPS variant as a Hailo-8, both at INT8, and states that both “use the memory on a Raspberry Pi 5”. The newer AI HAT+ 2 is a Hailo-10H at 40 TOPS INT4 with, in the documentation’s words, “its own 8 GB onboard memory”.

The workloads these run well are the convolutional vision family: detection, segmentation, pose estimation, classification. Arithmetic per byte read is high, the weights are small enough to stay resident, and the graph is static. That is the shape the hardware was designed around.

The large language model question

This is what most people arriving at “Raspberry Pi AI HAT” want to know, and the documentation answers it directly. Raspberry Pi’s comparison table marks large language model support as “Not supported” for both the 13 TOPS and 26 TOPS AI HAT+, and “Supported” only for the AI HAT+ 2, whose 8 GB of onboard memory it describes as “allowing it to run LLMs and VLMs up to ~6 billion parameters”.

The reason is the memory line, not the TOPS line. A language model at decode time is bandwidth-bound: it reads its entire weight set to produce each token. An accelerator with no memory of its own has to pull those weights across the host bus every step, and a single PCIe lane is orders of magnitude slower than the DRAM the Pi’s own CPU is already using. Giving the accelerator 8 GB of local memory is what changes the answer from “no” to “up to about 6B” — and 8 GB is also, not coincidentally, close to what 6B parameters at 4-bit plus a working context costs.

So a 13 or 26 TOPS HAT alongside a small chat model on the Pi does nothing for the chat model. The model still runs on the four Cortex-A76 cores at the speed the Pi’s own memory bandwidth allows, and the HAT sits idle unless you also have a vision pipeline for it.

What stays on the Pi’s CPU

Even for the vision workload these boards are built for, the accelerator is one stage in a pipeline and the rest is the Pi’s problem:

• Capture and decode. Pulling frames from a camera or demuxing and decoding a video stream happens before the accelerator sees anything.
• Pre-processing. Resize to the network’s fixed input size, colour space conversion, normalisation, layout change. This is per-frame work proportional to resolution, and at high frame rates it is frequently the actual bottleneck.
• Post-processing. Depending on how the model was compiled, decoding raw detection head tensors into boxes, and non-maximum suppression, may run on the CPU rather than on the chip.
• Your application. Tracking, business logic, storage, network. None of it is accelerated.

A pipeline that goes from 5 fps to 30 fps on the accelerator’s stage but spends 20 ms per frame resizing on the CPU has moved its bottleneck, not removed it. Profile the whole pipeline before attributing a disappointing frame rate to the accelerator.

There is also a power and thermal cost that arrives with the board. An accelerator drawing several watts through the Pi’s header or PCIe connector adds to a power budget that was already tight enough for Raspberry Pi to specify a particular supply, and undervoltage on a Pi presents as CPU throttling — that is, as the accelerator making the rest of your pipeline slower. Use the supply the documentation specifies and check vcgencmd get_throttled before concluding anything about performance.

One PCIe lane

Raspberry Pi’s own documentation describes the Pi 5’s expansion as a single-lane PCIe FFC connector. Everything an accelerator receives and returns crosses that one lane, and if you are also running an NVMe SSD from an M.2 HAT, it shares it.

For a vision model this is rarely the constraint: a 640x640 uint8 input tensor is about 1.2 MB, and 30 of those per second is well within a single lane’s budget. For anything that streams weights per inference it is fatal, which is the same mechanism that makes the on-board-memory distinction above decisive. It is also why a USB accelerator behaves similarly: the bus is the budget, and models that fit in on-chip memory are the ones that behave well.

How to decide before buying

Three questions, in order, and the first is disqualifying:

1. Does your model compile for the part? Hailo parts run graphs compiled by Hailo’s own toolchain into its binary format, from ONNX or TensorFlow. If your architecture is not in the vendor’s model zoo or its supported-operator set, the answer is not “it will be slower”, it is that there is no artefact to load. This is the same trap as the RK3588’s op set.
2. Is it a language model? Then only the part with its own memory is relevant, and the parameter ceiling in the vendor documentation is the ceiling.
3. Where is your time going now? Measure the current per-frame cost split between pre-processing, inference and post-processing. If inference is under half of it, an accelerator caps your improvement below 2x no matter how many TOPS it has.