Raspberry Pi and Single-Board AI: A Harness Instead of a Claim

Published numbers for single-board AI are unusable: they depend on the board revision, the memory variant, the operating system bitness, the power supply, the cooling and the ambient temperature, and almost none of those are stated. This page gives the arithmetic that predicts the result and a harness that measures it on the board in front of you.

What a Pi is genuinely good at

The successful projects share a shape, and it is not “a small server that runs a language model”.

• Low-frame-rate vision that runs forever. A camera watching a driveway, a production line or a bird feeder at one to five frames per second, running a small detector. The board is perfect for this and it costs a few watts.
• Audio classification and keyword spotting. Small models, continuous operation, no need for a display.
• Being the thing next to the sensor. Reading serial devices, GPIO, cameras and radios, doing the cheap local decision, and forwarding only what matters. This is the actual value proposition — the model is small and the point is that the raw data never has to be transmitted.
• Orchestration and glue. Running a queue, a small database, a local web interface, and calling a hosted model when a hard question arrives.

What it is bad at is interactive language model inference, and the reason is arithmetic rather than opinion.

The arithmetic that predicts token rate

Token generation is memory-bandwidth-bound: every token requires reading every active weight once. So the ceiling on tokens per second is bandwidth divided by model size, and nothing about the CPU changes it.

tokens_per_second_max = memory_bandwidth_bytes_per_s / weight_bytes

Worked, with B as the board's measured bandwidth in GB/s
and a 3B model at 4 bits (1.74 GB):

  B =  4 GB/s  →  2.3 tokens/s
  B =  8 GB/s  →  4.6 tokens/s
  B = 16 GB/s  →  9.2 tokens/s

And with a 1B model at 4 bits (0.58 GB):

  B =  4 GB/s  →  6.9 tokens/s
  B =  8 GB/s  → 13.8 tokens/s

This is a ceiling, not a prediction — real runtimes achieve some fraction of it, commonly half to three-quarters. But it settles the design question immediately: if the ceiling is three tokens per second, no amount of runtime tuning produces a conversational experience, and the correct response is a smaller model or a different architecture for the feature.

Deliberately, no bandwidth figure is asserted here for any board. Memory bandwidth varies by board generation and by memory variant, and the published theoretical peak is not what a real workload sees. Measure it.

The harness

Three measurements, in order: bandwidth, sustained token rate, thermal behaviour. Save the output next to your project, because the value of this is comparing your own configurations, not comparing to somebody else’s board.

#!/usr/bin/env python3
"""bench.py — measure what this board actually does.
Run with:  python3 bench.py  (nothing else running, board at rest)"""

import subprocess
import time

import numpy as np


def memory_bandwidth_gbps(size_mb=256, repeats=5):
    """Streaming copy bandwidth. Counts both the read and the write."""
    n = size_mb * 1024 * 1024 // 8          # float64 elements
    a = np.ones(n, dtype=np.float64)
    b = np.empty_like(a)

    np.copyto(b, a)                          # warm up, fault the pages in

    best = 0.0
    for _ in range(repeats):
        t0 = time.perf_counter()
        np.copyto(b, a)
        dt = time.perf_counter() - t0
        moved = a.nbytes * 2                  # read a, write b
        best = max(best, moved / dt / 1e9)
    return best


def temperature_c():
    """Pi-specific; returns None elsewhere."""
    try:
        out = subprocess.check_output(["vcgencmd", "measure_temp"], text=True)
        return float(out.strip().split("=")[1].rstrip("'C"))
    except Exception:
        try:
            with open("/sys/class/thermal/thermal_zone0/temp") as f:
                return int(f.read().strip()) / 1000.0
        except Exception:
            return None


def throttled_flags():
    """Pi-specific bitmask. Bit 0 under-voltage now, bit 1 ARM capped now,
    bit 2 throttled now, bit 3 soft temp limit now; bits 16-19 the same
    conditions having occurred at any point since boot."""
    try:
        out = subprocess.check_output(["vcgencmd", "get_throttled"], text=True)
        return int(out.strip().split("=")[1], 16)
    except Exception:
        return None


def sustained(run_once, seconds=300, label="workload"):
    """Run a workload repeatedly and report how it degrades over time."""
    t_end = time.time() + seconds
    rows = []
    start = time.time()
    while time.time() < t_end:
        t0 = time.perf_counter()
        units = run_once()                    # e.g. tokens generated
        dt = time.perf_counter() - t0
        rows.append((round(time.time() - start), units / dt, temperature_c()))
    print(f"\n{label}: elapsed_s, units_per_s, temp_c")
    for r in rows:
        print(f"  {r[0]:5d}  {r[1]:8.2f}  {r[2]}")
    first = rows[0][1]
    last = rows[-1][1]
    print(f"  degradation over {seconds}s: {(1 - last / first) * 100:.1f}%")
    return rows


if __name__ == "__main__":
    bw = memory_bandwidth_gbps()
    print(f"memory bandwidth (copy): {bw:.2f} GB/s")
    print(f"temperature at rest:     {temperature_c()} C")
    print(f"throttle flags:          {throttled_flags()}")

    model_gb = 1.74                            # your model's on-disk size
    print(f"token/s ceiling for a {model_gb} GB model: "
          f"{bw / model_gb:.2f}")

    # Then plug your own generation call into sustained():
    # sustained(lambda: generate_n_tokens(64), seconds=300, label="generate")

The bandwidth figure from a copy benchmark counts both the read and the write, which is the right convention for comparing against a workload that only reads — so treat the resulting ceiling as slightly optimistic and note that you did. Being explicit about the convention is what makes the number reusable.

Turning the numbers into a decision

Run the harness and you have four figures: measured bandwidth, the implied ceiling, the achieved rate, and the degradation over five minutes. Read them together rather than one at a time.

The last row is the point of the exercise. Discovering in twenty minutes that a board cannot sustain a conversational token rate is worth considerably more than discovering it after a fortnight of runtime tuning, and the arithmetic gives you the answer before you download a single checkpoint.

Power, heat and the throttling flags

The most common cause of a single-board computer being mysteriously slow is not the model. It is an inadequate power supply, and the board tells you so if you ask.

vcgencmd get_throttled

  0x0        healthy
  0x50000    under-voltage and throttling have occurred since boot
  0x50005    ...and are happening right now

Under-voltage causes the firmware to cap clocks silently. A supply that is adequate for an idle board is frequently not adequate for a board driving a camera, an SSD and every core at once, and the symptom is a benchmark that is inexplicably half of what a colleague measured. Check the flags before and after every run and record them alongside the result.

Thermal throttling is the other half. A board under sustained load without a heatsink will reach its limit within minutes and step down. The sustained() function above exists to measure exactly that: if your degradation figure is more than a few per cent, cooling is the cheapest performance improvement available to you and no software change competes with it.

One more environmental item that is easy to get wrong: run a 64-bit operating system. A 32-bit userland constrains per-process address space and misses optimised builds of the numeric libraries, and it costs more than most tuning gains.

Storage is the other bottleneck

Model loading is a large sequential read, and on a microSD card it is slow — often minutes for a multi-gigabyte file, and much worse if the card is a cheap one. Two consequences:

• Boot from USB or NVMe where the board supports it. This is usually the single largest improvement to perceived performance for an inference workload, and it is a hardware change rather than a tuning exercise.
• Memory-map the weights and keep the process alive. Loading once at startup and serving many requests turns a minutes-long cost into a one-off. A design that spawns a fresh process per request will spend nearly all of its time reading the same file.

Also plan for the card wearing out. A logging application that writes continuously to a microSD card will eventually kill it; put logs and any database on external storage, or in memory with periodic flushes.

When to add an accelerator

Add-on neural accelerators connect over USB or PCIe and change the arithmetic substantially for the models they support. Three things to establish before buying one:

1. Does it run your model, or a model like it? These devices typically execute a constrained integer operator set, and a model outside it either fails to compile or falls back to the host CPU. Compile your actual model with the vendor toolchain before committing.
2. Where does the data have to go? An accelerator on USB pays a transfer for every input and output. For small frequent inferences this transfer can dominate, which is why accelerators suit vision workloads with modest input sizes better than they suit token-by-token language generation.
3. What is the total power draw? Accelerator plus board plus camera plus storage against your supply’s rating. This is where the under-voltage flag starts appearing, and it is much easier to plan than to debug.

If the workload is a language model rather than vision, the honest answer is often that a board plus accelerator is the wrong shape entirely, and either a smaller model or a hosted call is better — the same trade-off examined in self-hosting versus calling an API, with the power supply as an additional term.