Skip to content
Multigrid

Edge and on-device AI

Running models on phones, laptops, single-board computers and microcontrollers, where memory, power and bundle size are hard limits rather than line items.

A model in a data centre is a cost problem. A model on a device is a budget problem, and the budgets are unforgiving: a fixed number of bytes you are allowed to hold resident, a battery that a user notices, a thermal envelope that silently halves your throughput in the second minute, and an app bundle that a store will reject.

The useful thing about budgets is that they are arithmetic. How much memory a model needs is parameters times bytes per weight, plus a KV cache you can calculate exactly, plus a working arena you can measure. How much energy a token costs follows from how many bytes must move to produce it. These pages derive those numbers in front of you, with every assumption labelled, so that when the hardware changes you can redo the sum rather than look for a new table.

Where a limit genuinely is device-specific — the per-app allocation ceiling, which operators run on which accelerator, what a store permits this quarter — the pages give the API or the command that reports it for your own target instead of a figure that would be stale before you read it.

What Actually Runs on a Phone: The Memory Arithmetic

Deriving the model size a phone can hold from parameters, bytes per weight, KV cache and the per-app allocation ceiling the OS enforces.

10 min read

Core ML and the Apple Neural Engine: Convert, Quantise, Profile

Converting a PyTorch model to Core ML, compressing the weights, and finding the operators that quietly fall back to CPU.

10 min read

On-Device AI on Android: Delegates, NPUs and Fragmentation

Running a model across thousands of Android SoCs by probing acceleration at runtime instead of guessing from device names.

10 min read

ONNX Runtime as a Portability Layer: Export, Verify, Deploy

Exporting a PyTorch model to ONNX, proving the numerics survived, and running the same graph across execution providers.

10 min read

TinyML on Microcontrollers: What Fits in 256 KB

The flash-versus-SRAM split, how to size a tensor arena from the widest layer, and what a 100 MHz part can actually infer.

9 min read

Quantisation for Edge Devices: What int8 and int4 Do to a Weight

The affine mapping behind integer quantisation, why the accuracy cost lands unevenly across layers, and what changes on integer-only hardware.

11 min read

Model Size Budgets for App Stores: Delivery Without a Rejection

The delivery mechanisms both stores provide for large assets, the review notes a downloaded model needs, and the egress cost nobody budgets.

9 min read

Battery and Thermal Cost of On-Device Inference: Joules Per Token

Deriving energy per token from bytes moved and energy per byte, then measuring it properly in your own app with the battery charge counter.

10 min read

Shipping a Model Inside a Desktop App: Packaging, Signing, Updates

Where a multi-gigabyte model should live relative to the signed bundle, how sandboxing constrains it, and how to update it without re-downloading everything.

11 min read

Wake Words and Always-On Audio: False Accepts and the Privacy Design

How a two-stage detector works, why false accepts are counted per hour rather than per inference, and what the ring buffer means for privacy.

9 min read

On-Device Vision: Detection, OCR and Segmentation Within a Frame Budget

Deriving the per-frame time and memory budget from the frame rate, and choosing model families that fit inside it.

10 min read

Offline-First AI Features: Degraded Modes That Do Not Lie

Routing between a local model and a hosted one, queueing what must wait, and telling the user which one answered without making it a burden.

10 min read

Federated Learning, Practically: The Protocol and What It Does Not Protect

What the federated averaging protocol actually does, which attacks secure aggregation stops, which it does not, and the deployment costs nobody mentions.

11 min read

Differential Privacy in Deployed Systems: What Epsilon Lets an Attacker Learn

The definition, epsilon translated into how much an attacker's belief can move, and the deployment costs the marketing leaves out.

11 min read

Personalisation Without Sending Data Home: Adapters, Caches and What Still Leaves

Where personal state actually lives when personalisation is local, and the parts of the system that still transmit something.

10 min read

Raspberry Pi and Single-Board AI: A Harness Instead of a Claim

What a single-board computer genuinely does well, the bandwidth arithmetic that predicts token rate, and a benchmark you run on your own board.

10 min read

Jetson and Edge GPU Boards: Power Modes, Toolchains and Deployment

Working within a fixed power envelope, why the toolchain versions are coupled, and why an optimised engine must be built on the target.

10 min read

Getting Camera and Sensor Data Into an Edge Model Without Falling Behind

Why an unbounded queue turns a throughput problem into unbounded latency, and how to drop frames deliberately instead.

10 min read

Updating Models on Devices You Do Not Control: Rollout, Rollback and Skew

Staged rollout by stable cohort, rollback that works without an app release, and keeping an old client model compatible with your current API.

10 min read

Protecting Weights You Shipped to Someone Else's Hardware

Why weights on a device you do not control are obtainable, what obfuscation actually buys, and where to put the value that cannot be copied.

10 min read

App Store Review for AI Features

What both stores actually check when an app can generate text or images: moderation evidence, age rating, billing for credits and data disclosures.

11 min read

Other topics