Updating Models on Devices You Do Not Control: Rollout, Rollback and Skew

Shipping a model to devices you do not control differs from a server deployment in one decisive way: you cannot roll everything back at once, and some fraction of your fleet will run the version you regret for months. Every part of the design follows from designing for that rather than hoping to avoid it.

What makes this different from a server deploy

• You cannot force anything. Devices are offline, asleep, out of storage, or owned by somebody who declines updates. Assume a long tail of old versions permanently.
• The artefact is large. A mistaken rollout costs real bandwidth on both sides, and the corrective rollout costs it again. The egress arithmetic in model size budgets for app stores is the reason a wrong rollout is expensive rather than merely embarrassing.
• Feedback is slow and thin. Server deployments give you full request logs within seconds. Device deployments give you whatever telemetry those devices choose to send, on their own schedule, minus everyone who has opted out.
• The model and the code version independently. A device can have new code with an old model, or old code with a new model. Both combinations must work, or must refuse to run clearly.

The manifest is the control surface

Serve a small, cacheable document that tells a client what it should be running. Everything else is a consequence of what this document says, which means you can change behaviour across the fleet without shipping code.

{
  "schema": 2,
  "models": [
    {
      "id": "summariser",
      "version": "2026-08-01-q4",
      "url": "https://cdn.example.com/models/summariser/2026-08-01-q4.bin",
      "sha256": "9f2c...",
      "bytes": 1739461632,
      "min_app_version": "4.2.0",
      "min_ram_mb": 4096,
      "rollout_percent": 10,
      "replaces": "2026-06-12-q4",
      "api_contract": "summariser/v3"
    }
  ],
  "kill": []
}

Each field earns its place:

• sha256 and bytes let a client verify a download and detect a truncated or tampered file before loading it.
• min_app_version and min_ram_mb stop a model reaching a device that cannot run it. The memory figure should come from the arithmetic in the phone memory budget, not from a guess.
• rollout_percent is the staging control, read by the client and applied to itself.
• replaces identifies what may be deleted once the new version has proven itself — never before.
• api_contract is the version-skew mechanism, and it is the last section of this page.
• kill lists versions that must not be used under any circumstances. A client finding its current version listed reverts immediately, without waiting for a download.

Staged rollout by stable cohort

The client decides whether it is in the rollout, using a stable hash of its own installation identifier. Stability is the whole point: the same device must get the same answer every time it asks, or raising the percentage will shuffle the population instead of extending it.

fun inRollout(installId: String, modelVersion: String, percent: Int): Boolean {
    if (percent >= 100) return true
    if (percent <= 0) return false

    // Salt with the model version so successive rollouts pick
    // different devices — otherwise the same unlucky 10% are
    // always the guinea pigs.
    val digest = sha256("$installId:$modelVersion")
    val bucket = ((digest[0].toInt() and 0xFF) shl 8 or
                  (digest[1].toInt() and 0xFF)) % 100

    return bucket < percent
}

Salting with the model version is the detail people miss. Without it, the same devices are first every time, so the same users absorb every bad release and your early signal comes from an unrepresentative population that has learned to distrust you.

1. Internal and opt-in testers first, at 100% for that group, identified by a flag rather than by the hash.
2. 1%, and wait for enough devices to have actually used the feature — not enough to have downloaded it. Downloads complete in hours; usage takes days.
3. 10%, then 50%, then 100%, with a hold at each step long enough to see the metrics that matter, which for on-device features is usually at least a few days.
4. Hold at less than 100% deliberately for a while. Keeping a few per cent on the previous version gives you a live control group, which is the only way to distinguish “the new model is worse” from “something else changed”.

Deciding whether it is going well

You need signals that arrive quickly and do not require reading user content. Report each one tagged with the model version, so that the comparison against the held-back cohort is direct.

Set the abort thresholds before starting the rollout, not while watching it. “Load success below 99%” or “inference errors above twice the control cohort” are decisions to make calmly in advance, and the same discipline as canary releases for model migrations on the server side.

Rollback that actually works

Rollback is a client capability, and it has to be built before it is needed.

1. Keep the previous model on disk until the new one has run successfully several times across separate sessions. Delete it on a schedule, not on first success.
2. Make reverting a manifest change, not a download. Setting rollout_percent to 0 and listing the version under kill should be sufficient for every device that still has the old file. Only devices that already deleted it need to fetch anything.
3. Roll back automatically on repeated local failure. If a model fails to load or crashes inference three times in a session, the client should revert on its own and report that it did. Devices that are offline cannot be told, so they must be able to decide.
4. Make the manifest itself resilient. Cache the last good manifest, apply a schema check before acting on a new one, and ignore a manifest that fails it. A malformed manifest that disables the feature fleet-wide is a self-inflicted outage with no rollback of its own.
5. Rehearse it. Roll a version back in a staging fleet deliberately, on a normal working day, before you need to do it under pressure.

Version skew against your own API

This is the part that is specific to on-device models and is almost never planned for. Your model runs on the device; your backend runs your code. They evolve on completely different schedules, and the model participates in a contract with the backend — an output schema, a set of labels, an embedding space, a prompt format.

The failures are concrete:

• Label sets change. A new local classifier emits a category your server does not know. Or an old client emits a category you have retired. Both must be handled, and the correct handling is to accept unknown labels and route them to a default rather than rejecting the request.
• Embedding spaces are not compatible. Vectors from model version N are meaningless against an index built with version N+1. If devices produce embeddings that your server stores or searches, the version must travel with every vector and the server must maintain both indexes through the transition — handling updates in a search index covers the mechanics.
• Output schemas drift. A field added on the device side arrives at a server that ignores it, which is fine. A field removed breaks a server that required it, which is not. Only ever add optional fields; never remove or repurpose one while any device might still send it.

The mechanism that makes this tractable is the api_contract field in the manifest. Every model version declares which server contract it speaks; every request from the device carries both its model version and its contract version; and the server keeps a compatibility matrix it can consult.

POST /v1/ingest
X-Model-Id: summariser
X-Model-Version: 2026-08-01-q4
X-Api-Contract: summariser/v3

Three rules make the matrix maintainable. Support at least the two previous contract versions, always. Refuse a contract you no longer support with a specific, actionable error rather than a generic 400 — the client can then prompt for an update instead of failing silently. And when you retire a contract, do it on a published schedule and check the telemetry first, because “nobody is on v1 any more” is a claim your own version-tagged metrics can either confirm or refute in about a minute.