What Actually Runs on a Phone: The Memory Arithmetic

A phone will run whatever fits in the bytes the operating system lets your process hold resident, at whatever speed the memory bus and the thermal envelope allow. That is one subtraction and one division, and doing them yourself is more reliable than any compatibility table, because the arithmetic does not change when the silicon does.

The number that decides everything

Not the neural accelerator. Not the total RAM printed on the spec sheet. The constraint is resident bytes for your process, and it is far below device RAM because the OS is defending every other app and its own compositor. Your budget looks like this:

resident_bytes  =  weight_bytes
                +  kv_cache_bytes(context_tokens)
                +  activation_arena
                +  runtime_and_app_overhead

must satisfy:   resident_bytes  <  per_process_ceiling

Three of those four terms are calculable before you write a line of code. The fourth — the ceiling — is the one you must ask the device for, and the section below names the calls that answer. Work the sum in that order and you will know which models are candidates before you download any of them.

Weights: parameters times bytes per weight

The weight term is exact. It is the parameter count multiplied by the average bytes each weight occupies on disk and in memory. The second factor is set by quantisation, and it is not the round number people quote.

Use the real figure, not the nominal one. The difference between 0.5 and 0.56 bytes is twelve per cent of your largest term, and twelve per cent is exactly the margin that decides whether a model loads or the process is killed. Worked:

4-bit, group size 32:
  packed weights   32 × 4 bits  = 16 bytes
  scale (fp16)                  =  2 bytes
  zero point (4 bits)           =  0.5 bytes
  ------------------------------------------
  per 32 weights                = 18.5 bytes
  per weight                    = 0.578 bytes  ( ≈ 4.6 bits )

1B params  → 0.58 GB
3B params  → 1.73 GB
8B params  → 4.63 GB

Embedding and output projection layers are often kept at higher precision than the body of the model, which adds back a few per cent on a small model and is negligible on a large one. If your checkpoint documents a mixed scheme, sum the layers rather than applying one factor to the whole parameter count. The general shape of this calculation is the same one behind server-side VRAM requirements; the difference on a phone is only that the ceiling is much lower and enforced much more abruptly.

The KV cache, which almost nobody budgets for

Every token already in the context keeps a key and a value vector in every layer, and those stay resident for the whole generation. This term grows linearly with context length and it is routinely larger than people expect:

kv_bytes_per_token = 2 × layers × kv_heads × head_dim × bytes_per_element

Worked, for a small model with grouped-query attention:
  layers          28
  kv_heads         8      (not the 32 query heads — GQA shares them)
  head_dim       128
  bytes_per_elem   2      (fp16 cache)

  2 × 28 × 8 × 128 × 2 = 114,688 bytes/token  ≈ 112 KiB/token

  4,096 tokens →  448 MiB
  8,192 tokens →  896 MiB

On a 3B model quantised to 4 bits — 1.73 GB of weights — an 8k context in an fp16 cache adds another 0.9 GB. The cache is a third of the footprint and it is the term that moves while your app is running.

Two levers matter. First, grouped-query attention: if that same model used 32 KV heads instead of 8, the cache would be four times larger — 448 KiB per token, 1.75 GiB at 4k. GQA is the single architectural choice that made long context on a phone possible at all. Second, quantising the cache to int8 halves the term for a quality cost that is usually smaller than quantising the weights by an equivalent amount, because the cache is transient. Check which your runtime supports before you plan a context length around it. The mechanism itself is the same one described in the KV cache explainer.

What else is in the budget

• The activation arena. Scratch space for the intermediate tensors of one forward pass. For a decoder generating one token at a time this is small relative to the weights — tens of megabytes — but during prefill of a long prompt it scales with the number of tokens processed per batch. Runtimes that chunk prefill let you trade prefill speed against this term directly.
• The tokenizer and the vocabulary. Modest, but not zero: a large vocabulary with a merge table is measured in megabytes, and it is easy to load twice by accident.
• Everything else your app is. Images, view hierarchies, a web view, a database. The model is a tenant, not the landlord, and on a phone the camera preview you left running is a real competitor for the same budget.

There is one structural trick worth knowing: memory-mapping the weights. If the weight file is mapped rather than read into an allocation, its pages are clean and file-backed, so under pressure the kernel can evict them and fault them back from storage instead of killing your process. The bytes still occupy physical memory while in use and the accounting differs between platforms, but a mapped model degrades under pressure where a heap-allocated one dies. Most on-device runtimes map by default; verify that yours does rather than assuming it.

Ask the device, do not consult a table

Per-process memory ceilings are not published as fixed numbers, they differ by device and OS version, and they are enforced by killing you. Both platforms expose a way to ask at runtime, and asking is the only approach that stays correct:

• iOS. os_proc_available_memory() returns the bytes remaining before your app is at risk of being terminated. Call it before allocating the model, not after. There is also an entitlement that raises the limit on capable devices — check the current name and eligibility in Apple’s developer documentation rather than trusting a blog post, since both have changed.
• Android. ActivityManager.getMemoryClass() and getLargeMemoryClass() describe the Java heap limit, which is the wrong number for a model: native allocations and mapped files are not on that heap. Use ActivityManager.MemoryInfo for system-wide pressure, respond to onTrimMemory callbacks by releasing the model, and treat the low-memory killer as a fact of life on entry-tier devices rather than an edge case.

Write the probe first. A ten-line function that reports available memory, then loads, then reports again tells you more about your real budget than any survey of device specifications, and it keeps telling you after next year’s hardware ships.

The second minute is slower than the first

A phone has no fan. Sustained inference raises the die temperature, the governor lowers clocks, and throughput falls — often substantially — without any error being raised. A benchmark that runs for ten seconds measures the boost state, which is not the state your user experiences during a long summarisation.

Measure it the only way that is honest: run the same fixed workload in a loop for several minutes with the device off charge and at a stable ambient temperature, and plot tokens per second against elapsed time. If the curve is flat you are bandwidth-bound and fine. If it steps down after a minute or two, that step is your real sustained rate and the number you should design the feature around. Both platforms also surface a coarse thermal-state signal to apps, which is the right trigger for reducing work — shorten the response, drop to a smaller model, or hand the request to a server — rather than pushing on and being throttled.

What the models that fit are good at

Work the arithmetic through and, on a mainstream phone in 2026, the models that comfortably fit alongside a real application are in the region of one to four billion parameters at four bits. That size does some things genuinely well and others badly, and the split is predictable:

• Reliable. Classification into a small closed label set. Extracting fields from text you already have. Rewriting, shortening and tone changes. Intent routing. Deciding whether a request needs a bigger model — which is the highest-value on-device job there is.
• Unreliable. Factual recall about the world; a small model has seen less and compressed it harder. Multi-step arithmetic and reasoning chains. Long-context recall, where the cache budget forces a short window anyway. Code generation beyond snippets.

The honest design is therefore usually a split, not a replacement: the local model handles the fast, private, offline-capable cases and escalates the rest. That pattern is what offline-first AI features is entirely about, and the escalation boundary is where most of the product design work actually lives.