ONNX Runtime Mobile, Start to First Inference

Getting an ONNX model running in a mobile app is straightforward. The part that takes the work is making the resulting binary small enough that nobody objects to it in review, and that is a build-time decision you make from your model.

Which package to add

The historical onnxruntime-mobile packages are gone; ONNX Runtime’s mobile documentation now points at the standard packages, which contain the full operator set and full ONNX format support:

• Android — onnxruntime-android, for Java, Kotlin, C and C++.
• iOS — the onnxruntime-c pod for C and C++, or onnxruntime-objc for Objective-C and Swift.
• .NET MAUI or Xamarin — Microsoft.ML.OnnxRuntime plus Microsoft.ML.OnnxRuntime.Managed.

// Android, app/build.gradle
dependencies {
    implementation 'com.microsoft.onnxruntime:onnxruntime-android:latest.release'
}

Start here even if binary size is your reason for reading. Get a correct inference with the full package first; the reduced build in the last section is derived from a model that already works, and debugging a missing-operator error inside a custom build you have never had working is a much worse afternoon.

Converting to ORT format

ORT format is a pre-optimised, flatbuffer-serialised form of your model. It loads faster because graph optimisations are applied ahead of time rather than at session creation, and — the important part — the conversion emits a configuration file listing exactly which operators and types the model needs. That file is the input to the reduced build.

pip install onnxruntime

python -m onnxruntime.tools.convert_onnx_models_to_ort ./models/

This produces a .ort file beside each .onnx, plus a configuration file in the documented format <operator domain>;<opset>;<op1>[,op2].... With type reduction enabled — ONNX Runtime 1.7 and later — the file also records the required types per operator and is named required_operators_and_types.config.

Convert every model your app will load, in one run, against the same directory. The configuration is a union, and a model you convert later and forget to include is a model whose operators are not in your reduced build.

First inference

1. Ship the model as an asset. Put the .ort file in app/src/main/res/raw/ on Android or in the app bundle on iOS. Loading from bytes rather than a path avoids extracting to temporary storage.
2. Create one environment and one session, and keep them. Session creation is the expensive step — it reads the model, allocates arenas and initialises providers. Creating a session per inference is the most common reason mobile ONNX Runtime looks slow.

   val env = OrtEnvironment.getEnvironment()
   val opts = OrtSession.SessionOptions()
   val modelBytes = resources.openRawResource(R.raw.model).readBytes()
   val session = env.createSession(modelBytes, opts)

3. Build the input tensor with the exact shape and dtype the model declares. Read them from the session rather than hard-coding: session.inputInfo gives the names and TensorInfo the shape and element type. A shape mismatch here produces an exception with the expected dimensions in it, which is the fastest thing in this whole page to debug.

   val shape = longArrayOf(1, 3, 224, 224)
   val buffer = FloatBuffer.allocate(1 * 3 * 224 * 224)
   // fill buffer with normalised NCHW pixel data, then:
   buffer.rewind()
   val input = OnnxTensor.createTensor(env, buffer, shape)

4. Run, and close what you allocate. The output map and the tensors are native resources; leaking them on a mobile device shows up as a memory graph that only climbs.

   session.run(mapOf(session.inputNames.first() to input)).use { results ->
       val output = (results[0].value as Array<FloatArray>)[0]
       // argmax, threshold, whatever the model is for
   }
   input.close()

5. Move preprocessing off the main thread. Resize, colour conversion and normalisation of a camera frame are per-frame CPU work, and on a mobile pipeline they are regularly larger than the inference itself.

Choosing an execution provider

ONNX Runtime’s mobile documentation lists CPU on every platform, NNAPI and XNNPACK on Android, and CoreML and XNNPACK on iOS. Its own recommendation is a sequence rather than a choice: start with CPU for a quantized model or XNNPACK for an unquantized one, and only move to a platform accelerator if you have a performance target you are missing.

The reason to be cautious about the accelerator providers is fallback. The NNAPI provider documentation notes that NNAPI may fall back to its own CPU implementation for operations the GPU or NPU does not support, and that this implementation “is often less efficient than the optimized versions of the operation of ORT”. The provider offers NNAPI_FLAG_CPU_DISABLED precisely so you can forbid that and find out. Its other documented flags are NNAPI_FLAG_USE_FP16, which allows reduced-precision relaxation, NNAPI_FLAG_USE_NCHW, which the docs warn may perform worse than NHWC, and NNAPI_FLAG_CPU_ONLY, which is for producing a reference output to validate against.

So the honest procedure is: measure CPU, measure XNNPACK, measure the accelerator with CPU fallback disabled, and compare all three on your model on your target devices. An accelerator that silently partitions your graph and runs half of it on a slow CPU path is a regression that looks like an optimisation.

Cutting the binary down

The full package carries every operator kernel for every opset. Your model uses a few dozen. A custom build strips the rest, and ONNX Runtime’s mobile documentation gives the scale of it: a custom build supporting ResNet50 reduced the Android library from 16 MB to about 3.9 MB uncompressed.

1. Use the config file from the ORT conversion — the one produced in the second section, not a hand-written list.
2. Build with the reduction flags. The documented ones are --minimal_build (drops ONNX format support, so the build loads .ort files only), --include_ops_by_config with your config file, --enable_reduced_operator_type_support for type reduction, --disable_ml_ops to drop the classical ML operator types, and --disable_exceptions, which replaces exceptions with logging and abort. Android and iOS have their own helper scripts, build_custom_android_package.py and build_and_assemble_apple_pods.py.
3. Re-run your inference test against the custom build. This is not optional. A reduced build is a build that fails on anything outside the config, and the failure surfaces at session creation with a missing-kernel error naming the operator — which is the good outcome, and the reason to have a test that loads every model the app ships.
4. Re-generate the config whenever a model changes. Re-exporting from a newer training framework can change the opset or introduce an operator you did not have. Wire the conversion into the build rather than doing it by hand once.

Two limitations to plan around: --minimal_build means the app can no longer load plain .onnx files, so any runtime model download must ship in ORT format; and --disable_exceptions turns recoverable errors into process termination, which is the wrong trade if your app loads models it did not build. For the alternative runtime on the same devices, see ExecuTorch, and for the quantization step that usually precedes any of this, quantizing ONNX models for edge targets.