Setting Up a Coral USB Accelerator for TFLite Inference

The Coral USB Accelerator will only execute a TensorFlow Lite model that has been fully integer-quantized and then compiled by a separate x86-only tool. Get either of those wrong and it still works — it just runs on your CPU while the stick sits there.

What the device requires

Google specifies the USB Accelerator at 4 TOPS of int8 throughput at roughly 0.5 W per TOPS. The constraints that matter more than the throughput figure are in Coral’s model requirements documentation:

• Fully 8-bit quantized. Every tensor, int8 or uint8. Float16 and dynamic-range quantization are not accepted. See quantizing a model for TFLite for the conversion.
• Constant shapes at compile time. No dynamic dimensions, and model parameters must be compile-time constants.
• Tensor rank limits. Tensors are 1-, 2- or 3-dimensional; a higher-rank tensor may only have size greater than 1 in its three innermost dimensions.
• Roughly 8 MB of on-chip SRAM caches model parameters, shared with the model’s executable code. Anything past that is streamed from host memory over USB on every inference.

The last one sets the size of model that makes sense here, and it is worth turning into a parameter count. At int8, one parameter is one byte, so roughly 8 MB of cache holds roughly 8 million parameters minus whatever the executable code takes. A MobileNetV2 at around 3.5 million parameters sits comfortably inside that with room for its code; a network with 20 million parameters does not, and the excess is streamed from host memory across USB on every single inference. That is the difference between a device that answers in single-digit milliseconds and one that is a slower way to use your CPU.

Which is also why this is a vision accelerator for MobileNet- and EfficientDet-class networks and nothing else. There is no path from a GGUF file or a transformer checkpoint to an Edge TPU that ends anywhere useful: the smallest interesting language model is two orders of magnitude past the cache, so every token would stream the entire weight set over USB.

Installing the runtime and library

1. Add Google’s apt repository. On the host that will do inference — a Raspberry Pi or any Debian machine:

   echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" \
     | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list
   curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
   sudo apt-get update

2. Install one runtime, not both. libedgetpu1-std runs the device at its reduced clock; libedgetpu1-max runs it faster and hotter. Coral’s documentation is explicit that you cannot have both installed at once.

   sudo apt-get install libedgetpu1-std

3. Unplug and replug the accelerator. The package installs a udev rule, and a device that was already attached will not pick it up. Use a USB 3.0 port — on USB 2.0 the transfer of input tensors becomes the bottleneck for anything above small input sizes.
4. Install PyCoral, and check your Python version first. Coral’s documentation states PyCoral supports Python 3.6 through 3.9. Current Raspberry Pi OS ships a newer interpreter than that, so sudo apt-get install python3-pycoral may have nothing to install against. Two documented ways round it: run inference inside a container pinned to a supported Python, or skip PyCoral entirely and load the Edge TPU delegate directly from the TFLite interpreter, which has no such ceiling.

   import tflite_runtime.interpreter as tflite
   
   interpreter = tflite.Interpreter(
       model_path="model_edgetpu.tflite",
       experimental_delegates=[tflite.load_delegate("libedgetpu.so.1")],
   )
   interpreter.allocate_tensors()

Compiling a model for the Edge TPU

This step does not happen on the Pi. Coral’s compiler documentation states the Edge TPU Compiler requires a 64-bit Debian-based x86-64 system and has not been available for ARM64 since version 2.1. Compile on a laptop, a CI runner or Google Colab, then copy the result across.

1. Install the compiler on an x86-64 Debian host, using the same apt repository as above:

   sudo apt-get install edgetpu-compiler

2. Compile your quantized model. Input is a .tflite file that is already fully integer-quantized:

   edgetpu_compiler mobilenet_v2_quant.tflite

   The output is written to the current directory as input_filename_edgetpu.tflite — here mobilenet_v2_quant_edgetpu.tflite. Use -out_dir to put it somewhere else.
3. Copy the compiled model to the device that has the accelerator attached, along with your labels file.

Running it

Coral’s own example is the shortest end-to-end check. With the PyCoral examples and test data present:

python3 examples/classify_image.py \
  --model test_data/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite \
  --labels test_data/inat_bird_labels.txt \
  --input test_data/parrot.jpg

The documented output prints five inference times followed by the classification — the first noticeably slower than the rest, then a stable figure. That first-run gap is not noise. It is the model parameters being transferred into the Edge TPU’s on-chip cache; every subsequent inference reuses them. Any timing you take from a single cold inference is measuring the transfer, not the model, which is the most common way people conclude the accelerator did nothing.

Confirm that the delegate is actually in use rather than assuming it. The tell is the model file: a graph compiled for the Edge TPU contains a custom operator called edgetpu-custom-op, and an interpreter created without the delegate will refuse to allocate tensors for it rather than running it slowly. So a compiled model that loads without error and produces plausible output is already good evidence. If you built a fallback path that loads the uncompiled model when the delegate is missing — sensible for a product, dangerous for a benchmark — log which branch you took, because the two are otherwise indistinguishable from the output.

Two operational notes. Coral’s documentation warns that the libedgetpu1-max runtime makes the device run hot enough to be uncomfortable to touch; on a fanless enclosure the standard runtime is usually the better default. And the accelerator is a USB device that can re-enumerate under thermal or power stress, which appears in a long-running process as an inference that suddenly fails on a device handle that was fine a second ago. A service that runs this for days needs to be able to re-open the interpreter, not just retry the call.

Reading the compiler log

The compiler tells you exactly how much of your model it took, and this is the output to keep. It reports the number of Edge TPU subgraphs, the total operation count, an operation log naming which ops mapped to the Edge TPU and which stayed on the CPU, and three memory lines in the form Coral’s documentation gives:

On-chip memory available for caching model parameters: 6.91MiB
On-chip memory used for caching model parameters: 4.21MiB
Off-chip memory used for streaming uncached model parameters: 0.00B

Two numbers decide whether this was worth doing. If the off-chip streaming figure is non-zero, parameters are crossing USB on every inference and your throughput is bounded by the bus rather than by the 4 TOPS. And if the subgraph count is above one, the graph was partitioned: Coral’s documentation warns that when the compiler hits an unsupported operation it splits the graph and everything after the split runs on the CPU, which “can potentially slow the inference speed by an order of magnitude”. A model that maps to one subgraph with zero off-chip streaming is the target; anything else is a model to change, not a setting to tune.