Skip to content
Multigrid

Hardware for local inference

Consumer GPUs, single-board computers and NPUs — what fits, what it costs, and which spec is the one that actually binds.

Buying hardware to run a model locally is a question with a numerical answer, and almost every page written about it gives a taste-based one instead. The reason is that the two figures people want — what fits, and how fast it goes — are not published by anyone. Vendors publish VRAM, bus width, memory clock, board power and lane count. Those are inputs. Nobody publishes the output, because the output depends on a runtime, a quantization format, a context length and a build.

So these pages do the arithmetic in the open. Capacity comes out of the parameter count and the bits per weight, and the bits per weight comes out of a published file size rather than out of the name of the format. Speed is given as a ceiling implied by memory bandwidth, which is a real and useful number precisely because nothing can exceed it — and every page that derives one also gives the command that measures what your own machine actually does. Where a figure is a vendor’s and can change next quarter, it is dated and marked.

What Fits on an RTX 4090's 24GB, by the Numbers

The VRAM budget of a 24GB card worked out line by line: weights from published GGUF sizes, KV cache from the model's own layer and head counts, and what is left.

10 min read

Local LLMs on a 12GB RTX 3060, by the Numbers

The realistic model ceiling for 12GB, derived from published file sizes — and the partial-offload cliff that decides whether a configuration is usable or unusable.

10 min read

Buying a Used GPU for Local LLM Inference

What to check on a second-hand card when the workload is inference rather than gaming — and why the usual used-GPU advice checks the wrong things.

9 min read

Running Two Consumer GPUs Without NVLink for Local Inference

How layer-split inference works when the only path between two cards is the PCIe bus, and why that is enough for single-stream generation.

10 min read

PCIe Lane Bandwidth in a Multi-GPU Local Setup, Derived

What a x4 slot costs compared with a x16 one, derived from the PCIe generation bandwidth tables — for model loading, for layer-split generation, and for prompt processing.

9 min read

Sizing a Power Supply for a Local LLM GPU Rig

Wattage headroom derived from published board power, the ATX 3.0 transient excursion limits, and the rest of the system — with every input named.

9 min read

Running a GPU Headless for Local Inference

Why a GPU with no monitor attached fails to initialise or runs slowly, starting from the exact error strings, and what to change.

9 min read

Local LLM Inference on a Laptop's Mobile GPU

Why a mobile GPU with the same name as a desktop card performs differently, derived from the two specs that actually diverge: memory bandwidth and the power envelope.

9 min read

Running a Local LLM on an eGPU Over Thunderbolt

What the Thunderbolt link costs for local inference: a slower model load, and almost nothing once the weights are resident in the enclosure's card.

9 min read

Picking a GPU by VRAM per Dollar for Local Inference

A dollars-per-gigabyte figure computed from dated launch prices and published memory sizes, plus the second axis that the ratio hides.

9 min read

Local LLM Tokens per Second on a Raspberry Pi 5, Derived

A token-rate ceiling for the Pi 5 derived from its published memory speed and a small model's byte count, with the command to measure the real figure.

9 min read

Jetson Orin Nano Memory Budget for Local Models, Derived

Which models fit the module's shared 8GB, derived from NVIDIA's published specification — and why the operating system is inside the budget on this device.

9 min read

Running Inference on the RK3588's NPU

What the RK3588's 6 TOPS neural engine will and will not execute, and the conversion toolchain every model has to survive before it touches the NPU.

10 min read

Setting Up a Coral USB Accelerator for TFLite Inference

Runtime, compiler and Python library for the Edge TPU, ending with a compiled model running through the USB stick and a compiler log you can read.

10 min read

Running a Small Chat Model on a Raspberry Pi

Build llama.cpp on a Pi, run a sub-2B GGUF model, and get a real tokens-per-second figure from llama-bench rather than from somebody's screenshot.

10 min read

AI Accelerator HATs for the Raspberry Pi

What an M.2 or HAT accelerator takes off the Pi's CPU, what stays on it, and which of the boards can run a language model at all.

9 min read

Local Inference on an Orange Pi 5, by the Numbers

What model sizes fit each Orange Pi 5 memory configuration, derived from published specifications, and why the board's 6 TOPS figure is not the limit you will hit.

10 min read

Running Larger Local Models on a Jetson AGX Orin

The model-size ceiling for both AGX Orin memory configurations, derived from NVIDIA's published capacity and 204.8 GB/s bandwidth figures.

10 min read

Running Ollama on a Raspberry Pi

Install Ollama on a 64-bit Pi, move the model store off the SD card, size the swap file, and configure the daemon so it survives a long generation.

10 min read

What Storing Model Files Does to an SBC's SD Card Over Time

Where the writes come from when a single-board computer runs local models, how write amplification multiplies them, and why the fix is moving the model store rather than buying a bigger card.

9 min read

Quantizing a Model for TFLite

Convert a float model to a fully integer-quantized .tflite file with a representative dataset, then verify the result against the original's outputs before you ship it.

10 min read

ONNX Runtime Mobile, Start to First Inference

Convert an ONNX model to ORT format, add the runtime to an Android or iOS project, run one inference, and then cut the binary down with a reduced build.

10 min read

Model Size Limits for WebGPU Inference in the Browser, Derived

The three separate ceilings on a browser-hosted model — buffer limits, device memory and the download — derived from the WebGPU specification's own default values.

10 min read

WebLLM: Start to First Response in the Browser

Load a quantized model into the browser's GPU with WebLLM and stream a completion, with the download cost stated up front.

9 min read

Running Inference on a Ryzen AI NPU

What the XDNA NPU in a Ryzen AI part actually accepts, which toolchain reaches it, and which half of an LLM it speeds up.

9 min read

Running Inference on an Intel Core Ultra's NPU

The OpenVINO path to the Core Ultra NPU, the static-shape constraint that shapes everything you can run on it, and where the TOPS figure comes from.

9 min read

On-Device Inference on Qualcomm's Hexagon NPU

Why the Hexagon HTP backend refuses a floating-point model outright, what a conforming model looks like, and where Qualcomm's 45 TOPS figure comes from.

9 min read

Quantizing a Core ML Model for iOS

Palettize a Core ML model with coremltools to cut its on-device footprint, then verify the compressed model against the fp16 original before you ship it.

9 min read

Bundling a Quantized Model Inside a Mobile App

What a bundled model actually costs in app-bundle bytes, and which App Store and Google Play limits it runs into first.

10 min read

Quantizing a Model for ONNX Edge Deployment

Take an ONNX model through onnxruntime's quantization toolchain to an int8 model, with the size compared before and after and the saturation trap avoided.

10 min read

What Happens When a Browser Doesn't Support WebGPU

Diagnose a browser that silently dropped WebGPU and ran your model on WebAssembly instead, and decide what to do about it deliberately.

10 min read

Why NPUs Use So Much Less Power Than GPUs for On-Device Inference

What TOPS-per-watt can and cannot be derived from published figures, and why it is the wrong number for single-stream LLM decode anyway.

11 min read

Running Inference on Windows With DirectML

Get an ONNX model running on any DirectX 12 GPU in Windows through the DirectML execution provider, without a vendor SDK — and know what Microsoft has moved to instead.

9 min read

ExecuTorch for On-Device PyTorch Inference

Export a PyTorch model to a .pte file with ExecuTorch, lower it to a mobile backend, and run it — with the export step that actually fails explained.

10 min read

Other topics