Offline-First AI Features: Degraded Modes That Do Not Lie

An offline-first AI feature is not a feature that works offline. It is a feature that has a defined behaviour for every combination of connectivity, local capability and request type — including the combination where the honest answer is “not now”.

Route on capability, not on connectivity

The usual implementation checks whether the device reports a network connection and branches. It is wrong in both directions. A device on a captive-portal wifi network reports connectivity and has none. A device with a working connection may still be the wrong place to send a request that must not leave the device. And a device that is offline may have no local model, in which case knowing it is offline does not help.

Route on a capability object instead — what can I do right now — and recompute it on the events that change it:

type Capability = {
  local: "ready" | "loading" | "unsupported";
  remote: "reachable" | "unreachable" | "unauthorised";
  metered: boolean;
  batteryLow: boolean;
};

remote is reachable only after a request has actually succeeded recently, not because an interface flag says so. Treat a failed request as evidence and mark it unreachable for a short backoff window — the standard circuit breaker pattern applies unchanged here, and it is what stops an offline device retrying a dead endpoint on every keystroke.

Three classes of request

Not every feature degrades the same way. Classify each one at design time and the routing rules write themselves.

There is a fourth case worth naming so it can be refused explicitly: requests that are neither degradable nor deferrable — the user needs this answer now and only a remote model can produce it. The correct behaviour is a clear, immediate, non-blaming message. What is not correct is a local model producing a worse answer presented as though it were the same answer, which is the failure this whole page exists to prevent.

The router

type Route =
  | { kind: "local" }
  | { kind: "remote" }
  | { kind: "queue" }
  | { kind: "refuse"; reason: string };

function route(cls: RequestClass, cap: Capability): Route {
  if (cls === "local-sufficient") {
    return cap.local === "ready"
      ? { kind: "local" }
      : cap.remote === "reachable"
        ? { kind: "remote" }
        : { kind: "refuse", reason: "model-not-ready" };
  }

  if (cls === "degradable") {
    // Prefer remote for quality, but not at any price.
    if (cap.remote === "reachable" && !(cap.metered && cap.batteryLow)) {
      return { kind: "remote" };
    }
    if (cap.local === "ready") return { kind: "local" };
    return { kind: "queue" };
  }

  // deferrable
  if (cap.remote === "reachable") return { kind: "remote" };
  return { kind: "queue" };
}

Two details in that function are the ones that matter in practice. Local-sufficient requests fall back to remote rather than failing when the local model is still loading, so a cold start does not break the feature. And degradable requests prefer remote but check metered connection and battery state first, because a user on a train should not be charged for data to get a marginally better summary.

Keep the two implementations behind one interface with one result type, including the field saying which produced it. If the call sites can tell the difference, the difference will leak into the UI inconsistently.

The queue, and idempotency

Anything queued must survive the app being killed, must not run twice, and must not run at all if the user has since changed the input.

1. Persist the request, not the closure. Store the input, the class, a client-generated identifier and a timestamp in local storage. A callback held in memory does not survive a process death, which is exactly when the queue matters.
2. Give every queued request an idempotency key — a UUID generated once at enqueue time and reused on every retry. Send it with the request so a retry after an ambiguous timeout cannot produce a second charged completion. Without this, a flaky connection costs money and produces duplicates.
3. Invalidate against the current state. Before sending, check that the document, message or setting the request was about still exists and still matches. A summary of a paragraph the user deleted two hours ago is noise.
4. Bound the queue and the age. Drop requests older than a threshold and cap the total. An unbounded queue that drains all at once when a device comes online is a self-inflicted burst on your own API.
5. Drain with jitter. If ten thousand devices regain connectivity when a network comes back, an unjittered drain is a synchronised spike. Randomise the delay.

Caching results, and syncing between devices

An offline-capable feature almost always wants a result cache, and the cache key is where the design decisions hide.

cache_key = hash(
  normalised_input,     // whitespace-collapsed, not the raw string
  task_id,              // which prompt or which head
  model_id,             // local-1b or gpt-mini or whatever ran
  model_version,        // changes invalidate everything below it
  relevant_settings     // tone, language, length preference
)

Including the model identity in the key is the part that matters and the part usually omitted. Without it, a result generated by the small local model is served forever afterwards to a device that is now online and could have had the better answer. With it, the two coexist and the upgrade path is a cache miss rather than a migration.

If results sync between a user’s devices, three rules keep it from going wrong:

• Sync results, not models. Weights are large, device-specific and reproducible from a download. Sending them between devices costs bandwidth for nothing.
• Make results content-addressed. If the key above is the identifier, the same input on two devices produces the same key and one device’s answer serves the other. This turns synchronisation into a shared cache rather than a merge problem, and it removes conflicts entirely for anything derived rather than authored.
• For anything the user edited, do not merge silently. A generated draft the user changed on their phone and separately changed on their laptop is a genuine conflict. Last-write-wins loses work and the user will not know it happened. Keep both and say so — the honesty rule that governs the rest of this page applies to sync as much as to degraded answers.

Bound the cache by size and by age, and drop it entirely on a model version change if the model identity is not in your key — which is a reason to put it there rather than an alternative to doing so.

Telling the user, without nagging

The design goal is that a user never mistakes a degraded answer for a full one, and is never made to feel the product is broken.

• Label the output, not the app. A small, permanent marker on results produced locally — “offline draft” — beats a banner across the screen. It attaches the caveat to the thing it is about and it survives scrolling.
• Offer the upgrade rather than performing it. When connectivity returns, show an unobtrusive “improve this” action. Silently replacing text the user may have already edited is worse than leaving it alone.
• Never spin indefinitely. A queued request gets a state the user can see and cancel. A spinner with no end is the most common way an offline path becomes a support ticket.
• Say what is unavailable and why, in one sentence. “Full analysis needs a connection — this will run when you’re back online” is complete. “An error occurred” is not.
• Do not announce the good case. When the local model is the right tool and its answer is genuinely good, there is nothing to disclose about degradation. Labelling every answer with its origin trains users to ignore the label, which is the one outcome that defeats the purpose.

Testing the offline path properly

Offline behaviour breaks quietly because nobody exercises it. Make it routine:

1. Put a switch in your debug build that forces each capability field independently. Testing “offline” via aeroplane mode covers one of several states you need.
2. Test the ugly middle states explicitly: connected but the endpoint returns 503; connected but every request takes forty seconds; connected then disconnected halfway through a stream; local model file present but corrupt.
3. Kill the process with items in the queue and relaunch. The queue must still be there and must not double-send.
4. Run the whole suite on a device with the local model unavailable — entry-tier hardware, or a user who cleared app data — because that path is the one nobody writes tests for.