Edge Inference: What Actually Fits on a Phone

On-device inference is usually discussed in terms of what a model can do. The prior question is what the device can hold, how fast it can read it, and what reading it costs in joules — three calculations, all of which can be done before a line of code is written.

The memory budget is smaller than the RAM

A phone with 8 GB of RAM does not offer 8 GB to a model. The operating system, the launcher, background services and the foreground application all hold memory, and mobile platforms enforce per-process limits and will terminate a process that exceeds them. A defensible planning assumption is that a well-behaved application can hold somewhere between a quarter and a third of total RAM resident, and less if it must survive backgrounding.

P_max = (usable_RAM - KV - runtime) / b

8 GB device, ~2.5 GB usable, 4-bit weights (b ~= 0.55):
P_max ~= (2.5 - 0.3) / 0.55 ~= 4.0e9 parameters

12 GB device, ~4 GB usable:
P_max ~= (4.0 - 0.4) / 0.55 ~= 6.5e9 parameters

So the practical class on mainstream phones is roughly 1B to 4B parameters at 4-bit, and the upper end of that is uncomfortable rather than routine. Memory-mapping the weights from storage instead of loading them helps startup and the memory accounting, but it moves the reads onto flash, which is far slower than RAM and turns the bandwidth calculation below into a much worse one.

There is a second-order cost that only appears on a device somebody is holding. A model resident in memory is memory unavailable to everything else, so a large one makes the rest of the system evict more aggressively and other applications reload more often. The user experiences that as the phone feeling slow, attributed to the phone rather than to your feature. Loading on demand and releasing promptly is usually the right design even though it costs startup latency, which is the opposite of the trade you would make on a server.

The bandwidth ceiling on a phone

Same bound as anywhere else. Mobile LPDDR bandwidth has sat in the tens of GB/s for mainstream devices in recent generations — an order of magnitude, and one you should replace with your target device’s published figure.

3B model at 4-bit:  P * b = 3e9 * 0.55 = 1.65 GB per token

at 50 GB/s   ->  ~30 tokens/second
at 25 GB/s   ->  ~15 tokens/second

A 1B model at 4-bit is 0.55 GB per token:
at 50 GB/s   ->  ~90 tokens/second

These are ceilings, and thermal throttling makes the sustained figure lower than the burst figure in a way that is much more pronounced on a passively cooled device than on a desktop. A phone that generates quickly for ten seconds and then slows is not malfunctioning; it is shedding heat it cannot dissipate.

Prefill is the other half and it behaves as it does everywhere: compute-bound, and mobile compute is modest. The practical consequence for interface design is that time to first token grows visibly with prompt length on a phone, so a long system prompt that costs nothing on a server is a user-visible delay here. It is also the argument for keeping a local model’s context genuinely short rather than merely within its limit — the KV cache is the memory term that grows, and the prefill is the latency term that grows, and both are worst exactly on the hardware least able to absorb them.

A lower bound on energy per token

Energy is where on-device inference is genuinely different, and it can be bounded from below without measuring a device — which is the only honest thing to do here, since actual battery drain depends on the screen, the thermal state, the scheduler and the specific silicon.

Moving data dominates. Published work on the energy cost of computation — Horowitz’s ISSCC 2014 survey is the canonical reference — puts off-chip DRAM access one to two orders of magnitude above on-chip SRAM access, and both far above the arithmetic itself. Using e for energy per byte read from DRAM:

E_token  >=  bytes_per_token * e

Assumption: e = 10 pJ/byte for LPDDR (order of magnitude;
substitute a figure for your memory if you have one).

3B at 4-bit: 1.65e9 bytes * 1e-11 J = 0.0165 J per token

A 15 Wh battery holds 15 * 3600 = 54,000 J.

54,000 / 0.0165 = ~3.3 million tokens

...as an ABSOLUTE LOWER BOUND on energy, ignoring compute,
memory controller, SoC static power, the display, and every
inefficiency. Real consumption is a multiple of this, and the
bound's value is in the ratio it establishes, not the total.

Two things fall out of that bound and both are useful. First, the energy scales with bytes moved, so quantisation is an energy optimisation of exactly the same factor as it is a speed optimisation — halving b halves both. Second, the arithmetic is not the problem; the memory traffic is, which is why mobile accelerator design is preoccupied with keeping weights on-chip and why small models are disproportionately more efficient than their parameter ratio suggests.

What the NPU changes

Mobile systems-on-chip include a neural processing unit: fixed-function matrix hardware with large on-chip buffers, designed for low-precision integer arithmetic. It improves performance per watt substantially for work it can accept, and the qualifier is doing real work.

• It does not add bandwidth. The NPU reads the same LPDDR as everything else, so the decode ceiling above is unchanged. What it improves is the joules and the compute-bound prefill.
• Its supported operations are narrower. Unsupported layers fall back to the CPU or GPU, and a model that falls back frequently can be slower than one that never used the NPU.
• It is usually quantised-integer first. Which suits on-device inference, since you were quantising anyway for the memory budget.
• Access is platform-specific. Reaching it means a vendor runtime, which is a portability cost paid per platform.

What the constraints leave you

The three calculations converge on a consistent picture: a 1–4B model, 4-bit, short contexts, generating at conversational speed, at an energy cost that permits bursts rather than continuous generation. That rules out a great deal and leaves a real set of jobs.

• Classification, extraction, routing and rewriting — short input, short output, and quality at this size is adequate for well-specified tasks.
• Anything where the data must not leave the device. This is usually the actual reason, and it is a reason that does not depend on the model being competitive with a hosted one.
• Offline operation and zero marginal cost. No network round trip, no per-token charge, no rate limit.
• A local first pass with escalation. Handle the easy majority on-device and route the rest to a larger hosted model. This is the pattern that gets the latency and privacy benefits without pretending a 3B model is a frontier one.