Core ML and the Apple Neural Engine: Convert, Quantise, Profile

Getting a model into Core ML is a converter call. Getting it to run on the Neural Engine is the actual work, and the reason it is work is that nothing tells you when it did not: an unsupported operator silently splits your graph, the unsupported part runs on CPU, and the only symptom is that the model is slower than it should be.

The pipeline, and where it usually breaks

1. Trace or export the PyTorch model to a form the converter accepts.
2. Convert to a Core ML package with coremltools, declaring input shapes and a minimum deployment target.
3. Compress the weights, then re-check accuracy on your own eval set.
4. Profile on a real device and find which operators did not run where you assumed.
5. Fix the graph, or accept the placement, and ship.

Step four is the one that gets skipped and the one that decides whether the exercise was worth doing. Steps one to three usually succeed on the first or second attempt; the model then runs at a fraction of the speed the hardware is capable of, and without a profile there is no signal telling you so.

Converting from PyTorch

The converter takes a traced module and a description of the inputs. Trace with a representative example, because tracing records the control flow that this input took — a model with data-dependent branching will be silently frozen into one branch.

import torch
import coremltools as ct

model.eval()
example = torch.randn(1, 3, 224, 224)
traced = torch.jit.trace(model, example)

mlmodel = ct.convert(
    traced,
    inputs=[ct.TensorType(name="image", shape=example.shape)],
    convert_to="mlprogram",
    minimum_deployment_target=ct.target.iOS17,
)
mlmodel.save("Model.mlpackage")

Two arguments matter more than they look. convert_to="mlprogram" selects the newer model type, which is the one that supports the compression tooling below; minimum_deployment_target decides which operator set the converter is allowed to emit, so raising it can turn a multi-operator workaround into a single supported operator, and lowering it can silently cost you performance.

If the model must accept variable input sizes, declare that explicitly rather than letting it be inferred. Core ML supports enumerated shapes and ranged shapes, and the two behave very differently on the accelerator: a small enumerated set can be specialised ahead of time, while an open range often cannot. Prefer an enumerated set of the three or four sizes you actually use.

Compressing the weights

Core ML supports several compression families, and they are not interchangeable:

Each of these exists in two modes: applied to an already-trained model, or applied during training so the model learns to tolerate it. The post-training form takes minutes and costs some accuracy. The training-time form costs a fine-tuning run and usually recovers most of it. Start with post-training, measure, and only pay for the training run if the measurement says you must — the reasoning behind that ordering is set out in quantisation for edge devices.

Whichever you choose, re-run your own evaluation afterwards. A compressed model that still passes a smoke test can have lost a specific capability entirely — the failure mode is narrow and it does not show up in average metrics.

Compute units are a request, not a promise

At load time you tell Core ML which processors it may use. The API is small and stable:

import CoreML

let config = MLModelConfiguration()
config.computeUnits = .all          // .all | .cpuAndGPU | .cpuAndNeuralEngine | .cpuOnly

let model = try MyModel(configuration: config)

This is a permission set, not an instruction. Core ML partitions the graph and places each partition on whichever permitted unit it judges best — and any operator the Neural Engine cannot execute forces a partition boundary. Every boundary costs a handoff, and a model chopped into six partitions can be slower than the same model pinned to one slower unit, because it pays five handoffs.

The constraints that cause boundaries are structural rather than arbitrary. The Neural Engine works in reduced precision and expects tensor layouts of a particular shape; dynamic shapes, unusual reduction patterns, control flow and custom operators are the recurring offenders. Which specific operators are affected depends on the OS version and the silicon generation, which is precisely why the next section measures instead of listing.

Finding the fallback: the differential method

There are two instruments and you want both.

The performance report

Open the .mlpackage in Xcode, choose the performance tab, and run it against a connected device. The report gives per-operator timing and, crucially, the compute unit each operator was assigned to. This is the direct answer and it should be your first stop. It requires a real device: the simulator has no Neural Engine and its numbers mean nothing.

The differential timing method

Where the report is unavailable or ambiguous, infer placement by timing the same model under three different permission sets:

for units in [.cpuOnly, .cpuAndGPU, .all] {
    let config = MLModelConfiguration()
    config.computeUnits = units
    let model = try MyModel(configuration: config)

    _ = try model.prediction(input: warmup)        // discard the first run
    let t0 = CFAbsoluteTimeGetCurrent()
    for _ in 0..<50 { _ = try model.prediction(input: sample) }
    let ms = (CFAbsoluteTimeGetCurrent() - t0) / 50 * 1000
    print("\(units): \(ms) ms")
}

Read the three numbers together:

• .all is close to .cpuOnly — the accelerator is barely being used. Something in the graph is forcing nearly everything to CPU. Look for dynamic shapes first.
• .all is much faster than .cpuAndGPU — the Neural Engine is doing real work. Good.
• .cpuAndGPU beats .all — the partitioning overhead exceeds the accelerator’s benefit. Either fix the graph or ship with the narrower permission set, which is a legitimate outcome.

Discard the first inference every time. It includes lazy setup and it is not representative. Report a median across runs rather than a mean, for the same reason percentiles beat averages everywhere else: one scheduling hiccup drags a mean past every inference the user actually experiences.

First load is not free

Core ML compiles a model for the specific device on first load and caches the result. That compilation can take seconds for a large model, it happens once per model version per device, and if you do it on the main thread during app launch your users will see it as a hang.

Do it deliberately: compile ahead of time off the main thread — the model-compilation API returns a URL to the compiled artefact, which you store and load from thereafter — and show honest progress while it happens. If your model arrives by download rather than in the bundle, the compile step belongs in the same background task as the download, and the feature stays disabled until both finish. That sequencing is part of the update path described in updating models on devices you do not control.

The shipping checklist

1. Convert with an explicit minimum_deployment_target and explicit input shapes. Record both in your build script, not in somebody’s notebook.
2. Evaluate the converted model against the PyTorch original on the same inputs before compressing anything. Conversion itself can change numerics.
3. Compress, then evaluate again on your own task-specific set.
4. Profile on the oldest device you support, not the newest. The newest hides the problem.
5. Measure sustained throughput over several minutes, not a ten-second burst.
6. Move the first-load compile off the launch path and off the main thread.