Running Inference on an Intel Core Ultra's NPU

Reaching the NPU in a Core Ultra laptop is one string in one function call. Everything difficult about it comes afterwards, from a constraint the plugin states plainly and most tutorials skip: only models with static shapes are supported.

One device string, one plugin

Intel’s NPU is reached through OpenVINO’s NPU plugin, and the plugin is selected the same way every other OpenVINO device is — by name.

import openvino as ov

core = ov.Core()
print(core.available_devices)   # e.g. ['CPU', 'GPU', 'NPU']

model = core.read_model("model.xml")
compiled = core.compile_model(model, "NPU")

results = compiled(input_tensor)

If NPU is not in available_devices, the cause is almost always the driver rather than the code. OpenVINO’s NPU device documentation states that the plugin needs an NPU driver installed on the system to execute a model, with drivers supplied by Intel on Windows and through a separate kernel-side driver on Linux. The documented platform support is Windows 11 64-bit and Ubuntu 22.04 with a 6.6 or newer kernel.

Three NPU generations, three TOPS numbers

“Core Ultra NPU” now covers three distinct blocks, and the gap between them is large enough that a page written for one is misleading about another:

• NPU 3720 in the first Core Ultra parts (Meteor Lake). This is the device OpenVINO’s plugin documentation names explicitly as its supported target.
• NPU 4 in Core Ultra 200V (Lunar Lake), which Intel quotes at 48 TOPS, within a platform figure of 120 TOPS that also counts the integrated GPU and the CPU.
• NPU 5 in Core Ultra Series 3 (Panther Lake), launched at CES in January 2026, which Intel quotes at 50 TOPS within a 180 TOPS platform figure.

The platform number is the one to be careful with. It is a sum of peak rates from three separate engines, and no single model gets all of it: a workload running on the NPU gets the NPU’s share and nothing else. Intel publishes the per-SKU figures on its product specification pages, and that is the only place worth reading them from, because they differ between SKUs within the same series (Intel ARK).

Static shapes, and why that hurts LLMs

The plugin documentation says it directly: only models with static shapes are supported on the NPU, and dynamic shapes and batching are not. For a vision or audio model this is barely a constraint — you were going to fix the input resolution anyway. For a language model it is the whole problem, because autoregressive decoding is built out of a sequence length that grows by one every step.

The way round it is to reshape the model to fixed buckets: a fixed prompt length for prefill and a fixed KV cache length for decode, with padding to fill them. OpenVINO GenAI does this for you for supported LLM architectures, which is why the recommended path to running a chat model on the NPU is through OpenVINO GenAI rather than through raw compile_model plus your own loop. The cost is real and worth stating: a fixed cache length means you pay for the whole window whether or not you use it, and exceeding it means recompiling for a larger bucket rather than growing gracefully. The general version of this trade-off is in how the KV cache works.

It is worth understanding why the constraint exists rather than treating it as an arbitrary limitation, because it tells you what else to expect. The NPU is not a general processor executing a kernel per operator; it is compiled to, ahead of time, into a schedule of tiles and buffers with fixed sizes. A shape that is not known at compile time is a schedule that cannot be produced. That same fact is why compilation is slow, why the result is cacheable, and why an unsupported operator forces a partition rather than a slower kernel.

The consequence for planning is straightforward. Encoder-shaped work — embeddings, classification, speech recognition, keyword spotting — has fixed shapes by nature and lands on this hardware cleanly. Open-ended generation has to be forced into buckets first, and the forcing costs both memory and flexibility. If you are choosing what to move to the NPU on a mixed workload, start with the encoder and leave the decoder where it is.

Precision and what the hardware computes in

The plugin accepts floating-point models in F32 and F16 and quantized models in U8 and INT8, including mixed FP16-INT8 graphs. The detail that changes how you think about it is that the hardware’s computation precision is FP16 — so handing the NPU an F32 model does not buy F32 arithmetic, it just means the conversion happens somewhere.

For weight compression on LLMs the usual target is 4-bit or 8-bit weight-only quantization applied with NNCF before compilation, which reduces the bytes that have to move per token rather than the arithmetic. That is the right lever for decode, for the reason set out in the power page: single-stream decode is bandwidth-bound, so fewer bits per weight is worth more than more operations per second. Related background on choosing the width is in choosing a quantization level.

Compilation is slow; caching is the fix

Compiling a model for the NPU is not free and it is not fast, which people discover as a multi-second stall on every application start. There are two independent caches and it is worth knowing which one you are relying on. UMD dynamic model caching is handled by the user-mode driver and is on by default. OpenVINO’s own model caching is opt-in and is enabled by setting a directory:

core.set_property({"CACHE_DIR": "./ov_cache"})
compiled = core.compile_model(model, "NPU")   # first call compiles, later calls load

One more moving part worth knowing about: from OpenVINO 2026.1 the plugin’s preferred compiler type changed to Compiler-In-Plugin, overridable through the ov::intel_npu::compiler_type property. If you are following a guide written against an older release and the behaviour differs, that is a likely reason. Nobody publishes compilation times for arbitrary models on arbitrary machines; measure yours with the OpenVINO benchmark_app tool, which reports compile time and inference throughput separately.

Two operational details follow from caching that catch people in production. A cache entry is keyed on the model, the device and the plugin configuration, so changing any of them silently invalidates it and the next start pays full compilation again — which looks like a mysterious intermittent stall rather than what it is. And the cache directory has to be writable by the process at run time, which on a packaged desktop application usually means somewhere under the user’s application data rather than beside the executable.

Finally, a note on what to expect from the whole exercise. There is no published table of throughput for a given model on a given Core Ultra NPU, and there could not usefully be one, because the answer depends on the driver, the OpenVINO release, the shape buckets you compiled for and what else on the system is competing for memory bandwidth. The number that matters is the one benchmark_app prints on the machine you are shipping to, run against the same compiled artefact your application will load.