Running Inference on Windows With DirectML

DirectML is the answer to a specific question: how do I run an ONNX model on whatever GPU this Windows machine has, without shipping CUDA for NVIDIA, ROCm for AMD and something else for Intel. It answers that question well. It is also no longer where Microsoft is putting new work, and you should know that before you build on it.

Read this before you start

Microsoft Learn’s introduction to DirectML now opens with an important notice: DirectML is in sustained engineering, and new feature development has moved to Windows ML for Windows-based ONNX Runtime deployments. The same notice says Windows ML provides the same ONNX Runtime APIs while dynamically selecting the best execution provider for the hardware (Microsoft Learn).

“Sustained engineering” means supported, shipped with Windows, and receiving security and compliance fixes — not removed, and not broken. The onnxruntime-directml package is still being released; version 1.24.4 was published in March 2026 for Windows x86-64 on Python 3.11 through 3.14 (PyPI). So the decision is straightforward: if you are starting something new and can require Windows 11 24H2 or later, look at Windows ML first. If you need to support older Windows, or you already have a DirectML path, this page is how it works.

What DirectML gets you, and what it does not

DirectML is a hardware abstraction layer over Direct3D 12. Its coverage claim is precise and worth quoting the shape of: it is supported by all DirectX 12-compatible hardware. That is the real value here — one binary that runs on NVIDIA, AMD, Intel and Qualcomm GPUs in Windows with no vendor SDK installed and no per-vendor build.

What it does not uniformly give you is the NPU. DirectML’s contract is with DX12 devices; reaching a Copilot+ class NPU is what Windows ML’s execution-provider selection is for, and what the vendor stacks do directly — AMD through the Vitis AI EP as described in running inference on a Ryzen AI NPU, Intel through OpenVINO’s NPU plugin in the Core Ultra page, Qualcomm through the QNN EP. If your requirement is specifically “use the NPU”, DirectML is not the route. If it is “use whatever GPU is here”, it is a good one.

Getting a model running

1. Install the DirectML build of ONNX Runtime. It is a separate wheel and it conflicts with the default one, so install it into a clean environment: pip install onnxruntime-directml.
2. Have an ONNX model with fixed or well-behaved shapes. If it came from PyTorch, export it with torch.onnx.export first.
3. Create the session with DmlExecutionProvider and the two session options it requires.
4. Verify with get_providers() that you actually got it, then run.

import numpy as np
import onnxruntime as ort

opts = ort.SessionOptions()
opts.enable_mem_pattern = False                       # required by the DML provider
opts.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL  # required by the DML provider

session = ort.InferenceSession(
    "model.onnx",
    sess_options=opts,
    providers=["DmlExecutionProvider", "CPUExecutionProvider"],
)

print(session.get_providers())
# ['DmlExecutionProvider', 'CPUExecutionProvider'] — if DML is missing,
# you are on the CPU and everything below is meaningless.

name = session.get_inputs()[0].name
x = np.random.rand(1, 3, 224, 224).astype(np.float32)
out = session.run(None, {name: x})[0]
print(out.shape)

The get_providers() check is the whole reason this tutorial does not end at the previous line. ONNX Runtime treats the provider list as a preference and drops what it cannot initialise, which on this platform means a missing DirectML runtime or a driver that does not expose DX12 quietly leaves you on the CPU. The browser version of exactly this failure is dissected in the silent-fallback page.

The two session options it requires

These are not tuning knobs, and the ONNX Runtime documentation states them as requirements rather than suggestions (ONNX Runtime DirectML EP docs). Memory pattern optimisation must be disabled, and the execution mode must be sequential. Both exist because DirectML manages Direct3D 12 resources itself and ONNX Runtime’s own arena and parallel scheduling assume they own allocation and ordering.

Leaving them at the defaults does not always fail immediately, which is what makes it dangerous — you get a session that works on your machine and produces allocation errors or wrong results elsewhere. Set both, every time, in the same block as the provider.

Choosing an adapter, and threading

On a laptop with both an integrated and a discrete GPU there are two DX12 adapters, and DirectML picks by index. The C# API takes it as an argument to AppendExecutionProvider_DML(0), where the value corresponds to the enumeration order of hardware adapters and 0 is always the default adapter. In Python the same choice is passed as a device_id provider option:

session = ort.InferenceSession(
    "model.onnx",
    sess_options=opts,
    providers=[("DmlExecutionProvider", {"device_id": 1}), "CPUExecutionProvider"],
)

The adapter enumeration is a system property and not stable across machines, so hard-coding an index is a bug waiting for a user with a different laptop. Enumerate and choose, or expose it as a setting.

Which adapter you want is also not always the obvious one. On a machine with a discrete GPU, the discrete part has its own memory and far more of it, which is what you want for anything large. On a machine with only integrated graphics, the GPU shares system memory with everything else, so a model that fits at all may still be contending with the rest of the system for bandwidth — the same effect that governs decode speed on an NPU, worked through in the power page. And on a laptop, choosing the discrete adapter is choosing to run the fans and drain the battery, which for a background feature is often the wrong call even when it is faster.

Finally, the threading rule, which is the one that bites in a web service. Only one thread may call Run at a time on a single DirectML-based inference session; multiple threads may call Run concurrently only if they operate on different session objects. If you are wrapping this in an HTTP handler, that means a lock around the session or a pool of sessions, and a pool of sessions means a copy of the weights per session in GPU memory — which is usually the reason to serialise instead. The general version of that trade-off is in continuous batching.