On-Device Inference on Qualcomm's Hexagon NPU

The Hexagon NPU is the least forgiving of the three PC NPUs, and it is the most honest about it. Its backend does not quietly fall back when you hand it something it cannot run: it refuses. Knowing exactly what it will accept turns a week of confusing failures into an export checklist.

What Qualcomm publishes

Qualcomm’s own Snapdragon X Elite product brief (document DCN 87-71417-1 Rev C) gives an SKU table in which all three listed parts — X1E-84-100, X1E-80-100 and X1E-78-100 — carry a Hexagon NPU rated at 45 NPU TOPS, alongside an Adreno GPU rated at 4.6 or 3.8 TFLOPs and LPDDR5x memory at a transfer rate of 8448 MT/s (Qualcomm product brief).

Notice that the NPU and the GPU are quoted in different units on the same table, and that neither carries a stated precision. TOPS counts integer operations; TFLOPs counts floating-point ones. They are not comparable, and the ratio between them says nothing about which engine will run your model faster. The one figure on that table you can do arithmetic with directly is the memory transfer rate, and that turns out to be the number that governs LLM decode — the derivation is in the power page.

The contract the HTP backend enforces

Most people meet the Hexagon NPU through ONNX Runtime’s QNN Execution Provider, and its documentation states the constraints without hedging (ONNX Runtime QNN EP docs):

• Quantized models only. The QNN HTP backend supports quantized models and nothing else. An fp32 or fp16 graph does not run slower on the NPU — it does not run on it at all.
• QDQ format. The model must carry explicit QuantizeLinear and DeQuantizeLinear pairs around the operators to be quantized, which is what lets the EP recognise a quantized subgraph and fuse it.
• uint8 weights with uint16 activations, or uint8 for both. The 16-bit activation option exists because activations have a much wider dynamic range than weights, and crushing them to 8 bits is where most of the accuracy loss on a transformer comes from.
• No dynamic shapes. The EP does not support models with dynamic dimensions, including a dynamic batch size. Every dimension must be fixed before export.

That last one and the quantization requirement together are why the NPU is a much better fit for a fixed-shape encoder — an embedding model, a wake-word detector, a speech encoder, an image classifier — than for an open-ended chat model. Fixing the shapes of a decoder means committing to a prompt-length bucket and a cache length in advance.

Getting a model onto it

The Python packaging reflects a two-machine workflow, and mixing them up is a common early mistake:

# On an x64 development machine: quantize.
pip install onnxruntime-qnn
python -m onnxruntime.quantization.preprocess --input model.onnx --output model-infer.onnx

# On the Snapdragon device (ARM64): run.
pip install onnxruntime-qnn

Once you have a QDQ model, the EP is selected with a provider option that names the backend library. This is the string that decides whether you are on the NPU or on the CPU reference backend, and getting it wrong is the classic “it runs but it is slow” outcome:

import onnxruntime as ort

session = ort.InferenceSession(
    "model-qdq.onnx",
    providers=["QNNExecutionProvider"],
    provider_options=[{"backend_path": "QnnHtp.dll"}],   # libQnnHtp.so on Linux
)
print(session.get_providers())   # confirm QNNExecutionProvider is actually present

The documented package requirements are specific enough to break a build: Python 3.11.x, with NumPy 1.25.2 or 1.26.4 and newer, the x64 package used for quantization and the ARM64 package used for on-device inference. Checking get_providers() after session creation is not optional here, for the same reason it is not optional in the browser — ONNX Runtime drops an unavailable execution provider and continues with whatever is left. That failure mode is dissected in the silent-fallback page.

Context binaries and the cold-start cost

The HTP backend compiles a graph into a device-specific binary before it can run it, and that compilation is slow enough to dominate a short session. ONNX Runtime supports caching the result as a context binary, controlled through session configuration entries: ep.context_enable set to "1" turns generation on, and ep.context_embed_mode set to "1" embeds the compiled binary inside the model file rather than beside it.

The trade-off is portability. A context binary is compiled for a specific HTP architecture version, so a binary generated for one Snapdragon generation is not guaranteed to load on another. If you ship the binary inside your app to avoid a first-run stall, you are shipping a per-device-family artefact, and you need a fallback path for the devices you did not anticipate. That interacts directly with how much you are willing to put in the app bundle — see bundling a quantized model inside a mobile app.

Where the NPU stops being the answer

Two limits are worth internalising before you invest in this path. The first is operator coverage: anything the HTP backend does not support falls back to CPU, and a fallback in the middle of a graph costs two tensor transfers per occurrence. The second is that the NPU does not have its own memory. It reads weights over the same LPDDR5x interface as everything else, so a decode loop that has to stream gigabytes of weights per token is bounded by that interface no matter how many TOPS the datasheet claims.

There is a third limit that is about people rather than silicon, and it is the one that decides whether this path is worth taking. The Hexagon toolchain is strict in a way that front-loads all of the effort: you cannot iterate by running the model and seeing what happens, because a non-conforming model does not run at all. Every change to the architecture means re-quantizing, re-validating the accuracy against the float reference, and regenerating the context binary. That is a good trade for a model you will ship for two years and a bad one for a model you are still designing.

Qualcomm publishes no tokens-per-second figures for arbitrary models, and any you find in a blog post are one device at one thermal state on one SDK version. Measure with qnn-net-run from the QNN SDK, which reports per-graph execution time on the device itself, or read the throughput ONNX Runtime reports for your own session. That is a number about your build; a number from someone else’s is not.