Running Inference on a Ryzen AI NPU

A Ryzen AI laptop has three engines that can run a model: the CPU cores, the RDNA integrated GPU, and an XDNA NPU that only one software path reaches. Almost everything confusing about this hardware comes from that last clause.

The 50 TOPS figure, and what it counts

AMD’s press release of 6 January 2025 announcing its CES AI PC portfolio quotes “up to 50 TOPS of AI processing ability” for the XDNA 2 NPU in the Ryzen AI Max series, and “leading peak 50+ NPU TOPS” for the Ryzen AI 300 series (AMD Investor Relations). That release does not state the precision the figure is measured at; AMD’s architecture disclosures for XDNA 2 describe the same engine as delivering its peak rate at INT8 and at a block floating-point format.

Two things follow, and both are the kind of thing that gets lost when the number is copied into a comparison table. First, a TOPS figure is a peak multiply-accumulate rate, not a throughput guarantee for any model. Second, it is not comparable across vendors unless the precision matches, and it usually is not stated. Microsoft’s Copilot+ certification bar of 40 NPU TOPS is why every vendor now quotes a number just above it; that is a procurement threshold, not a performance model.

The stack that reaches the NPU

The NPU is not a device your existing PyTorch or ONNX Runtime CPU code will find. AMD ships Ryzen AI Software, currently version 1.8.0, and inside it the Vitis AI Execution Provider is the component that decides which parts of an ONNX graph go to the NPU and which fall back to CPU. Above that sit three interfaces documented in AMD’s Ryzen AI Software documentation: the Lemonade Python API, a Lemonade server that exposes an OpenAI-shaped HTTP endpoint, and the native ONNX Runtime GenAI and llama.cpp APIs underneath.

The Lemonade server is the one to reach for if what you want is a local endpoint rather than an embedded runtime. It gives you the same request shape you already use, which means the difference between a hosted model and this laptop’s NPU is a base URL. The Python API and the native paths are for the case where the model is part of an application rather than a service.

What is actually supported

This is where most of the wasted afternoons live, so it is worth being precise. From AMD’s installation instructions for 1.8.0:

• Windows 11, build 22621.3527 or newer. Model generation is not supported on Linux in this release, although models generated on Windows are stated to be compatible with Linux.
• NPU driver 32.0.203.280 or newer, installed separately from the SDK by running npu_sw_installer.exe as administrator. A stale driver is the most common cause of a device that is present in Device Manager and invisible to the runtime.
• Supported silicon by AMD codename: Phoenix, Hawk Point, Strix, Strix Halo and Krackan Point — which is to say Ryzen AI 300 and Ryzen AI Max 300 parts, plus the earlier Ryzen 7000 and 8000 series parts that shipped an NPU.
• The installer creates a conda environment; the documented verification step is running quicktest.py from the quicktest folder inside the install directory.

What the NPU will accept

The NPU is an integer-first dataflow engine. AMD’s documentation describes the normal path as quantizing model parameters from floating point to lower precision such as 8-bit integer, and describes floating-point models as being internally converted to bfloat16. In practice that means you do not hand it an fp32 checkpoint and hope: you produce a quantized ONNX model, and the Vitis AI EP partitions the graph so that operators it supports at that precision run on the NPU and everything else runs on the CPU.

The partitioning is the thing to watch. A graph with an unsupported operator in the middle does not fail — it splits, and the tensor crosses between NPU and CPU twice per occurrence. That is how a model that “runs on the NPU” ends up slower than the same model on the CPU alone. The general shape of this problem is covered in edge quantization; the Ryzen-specific part is that the EP reports its partitioning, and reading that report is the first debugging step, not the last.

Which half of an LLM this speeds up

For a language model, an NPU helps the two phases of generation very differently, and this is mechanism rather than benchmark. Prefill processes the whole prompt in parallel and is bounded by arithmetic throughput, which is exactly what a 50 TOPS engine is. Decode produces one token per pass and must stream the model’s weights out of system memory to do it, so it is bounded by memory bandwidth — and on a laptop the NPU shares that bandwidth with everything else. Adding arithmetic to a bandwidth-bound loop does not make it faster.

This is why AMD’s own LLM paths use the iGPU and the NPU together rather than the NPU alone, and why a long-prompt workload (summarising a document, classifying a batch of records) sees a different picture from an interactive chat. If your workload is convolutional or encoder-shaped — vision, speech, embeddings — it is arithmetic-bound throughout and the NPU is doing the thing it was built for. The arithmetic behind that claim is worked through in why NPUs use so much less power than GPUs.

Nobody publishes tokens per second for a given model on a given Ryzen AI part, and any figure you find is a measurement of one machine at one driver version. The number you want is the one from your own hardware: run the model through the Lemonade server and read the usage statistics it returns, or use the throughput output of AMD’s own example scripts.