Converting a Hugging Face Model to ONNX

An ONNX export is not a file conversion. It traces the model with an example input, records the operators that ran, and writes that graph. Anything the trace did not see is not in the file — which is why the export prints a numerical comparison at the end and why that comparison is the part to read.

What an export actually produces

A safetensors checkpoint is weights plus a config; the code that turns them into a computation lives in the transformers package. ONNX removes that dependency by writing the computation itself: a graph of standardised operators with the weights as initialisers, which any conforming runtime can execute without the original Python.

The cost is that the graph has to be discovered by running the model. Python-level control flow that depended on the example input is baked in as whatever branch was taken. Dynamic shapes survive only where the exporter was told to mark an axis as dynamic — which Optimum does for batch and sequence length by default, and which --no-dynamic-axes turns off. A model with genuinely data-dependent structure either exports wrong or does not export.

What you get in return is portability of a specific kind. The same graph runs under ONNX Runtime on CPU, under its CUDA or DirectML execution providers on a GPU, under vendor toolchains that ingest ONNX, and in a browser through WebAssembly — without shipping Python. That is the reason to do this at all, and it is worth being clear that it is the reason: an ONNX export is rarely faster than PyTorch on the same hardware, and choosing it for speed alone usually disappoints.

Running the export

Optimum wraps the exporter with per-architecture configurations, which is what makes this one command rather than a scripting exercise.

pip install "optimum[onnx]"

# from the Hub; the task is inferred from the repository metadata
optimum-cli export onnx --model distilbert-base-uncased-distilled-squad \
  distilbert_squad_onnx/

# from a local directory: --task is required, because there is no Hub
# metadata to infer it from
optimum-cli export onnx --model ./my-model --task text-classification \
  ./my-model-onnx/

Three flags are worth knowing before you need them. --opset pins the ONNX operator set version, which you set when a downstream runtime supports only up to a particular version. --dtype takes fp32, fp16 or bf16. --atol overrides the tolerance used in the validation step below. The full list is in Optimum’s ONNX export guide, and it moves — the exporter gained a --dynamo option to select PyTorch’s newer export path, and the package itself has been split out of the main optimum distribution.

The validation block is the deliverable

After writing the graph, the exporter runs both models on the same dummy input and compares every output. This is the verification step that the row asks for, and it is already built in:

Automatic task detection to question-answering.
Framework not specified. Using pt to export the model.

Validating ONNX model...
        -[✓] ONNX model output names match reference model (start_logits, end_logits)
        - Validating ONNX Model output "start_logits":
                -[✓] (2, 16) matches (2, 16)
                -[✓] all values close (atol: 0.0001)
        - Validating ONNX Model output "end_logits":
                -[✓] (2, 16) matches (2, 16)
                -[✓] all values close (atol: 0.0001)
All good, model saved at: distilbert_squad_onnx/model.onnx

1. Check the output names line. A mismatch here means the graph exposes different outputs than the PyTorch model, and downstream code that indexes outputs by name will break.
2. Check each shape pair. A shape that matches only at the dummy batch size means an axis was not marked dynamic.
3. Check the tolerance actually used, printed in the atol parenthesis. A default that was loosened for your architecture is a signal, not a formality.
4. Then run your own comparison on real inputs, not the dummy one. Load the export with the matching ORTModelFor... class and compare logits against the PyTorch model on inputs from your actual distribution.

Step four exists because the built-in check uses generated dummy input of one shape. It proves the graph is not catastrophically wrong; it does not prove the export handles the long, ragged, real inputs your application sends.

Models over 2 GB

ONNX serialises with protocol buffers, and protobuf has a hard 2 GB limit on a single message. Any model whose weights exceed that cannot live in one .onnx file, and you meet the limit as:

ValueError: Message onnx.ModelProto exceeds maximum protobuf size of 2GB

The resolution built into the format is external data: the .onnx file keeps the graph structure and the initialisers move to a sibling file, conventionally model.onnx_data. Optimum does this for you when the export needs it, which is why a large export produces two files and why moving only the .onnx one gives a model that will not load. The ONNX external data documentation sets out the constraint: the data file must sit alongside the model, and tools such as the checker need the model’s path rather than a loaded proto to find it.

A 7B model in fp16 is about 14 GB and crosses this line by a wide margin, so for any decoder of interesting size the two-file layout is the normal case rather than an edge case. Treat the pair as one artefact when you copy, archive or containerise.

Decoders and the past-key-values suffix

For generative models the default export includes cache reuse, which Optimum expresses as the -with-past task suffix — text-generation-with-past rather than text-generation. The exported graph then takes past keys and values as inputs and returns updated ones, so the runtime can avoid recomputing attention over the prefix on every step.

Export without it and every generated token reprocesses the whole sequence, which turns generation from linear into quadratic in output length. If you have exported a decoder and found it far slower than the PyTorch original, this is the first thing to check — the mechanism is the same one described on the KV cache page, and an export that drops it drops the single largest optimisation in autoregressive decoding. Encoder-decoder models split into separate encoder and decoder graphs for the same reason: the encoder runs once and the decoder loops, and --monolith forces them back into one file at the cost of that separation.