Skip to content
Multigrid

Running local models day to day

Serving, securing, benchmarking and living with a local model — the operational questions that come after it loads.

Getting a model to load is the easy half, and it is the half every guide covers. The questions that follow are the ones that decide whether you keep using it: which of nine files in the repository you should have downloaded, why the same model that felt instant in a demo takes four seconds a paragraph on your machine, how much of the disk you have quietly given to three copies of the same weights, and what to do when the runtime refuses to load something it loaded yesterday.

These pages answer those. Where a number would help, it is either taken from a primary source and named in the sentence, or derived here from inputs you can check — a published memory bandwidth, a file size in a repository listing, a parameter count. Nothing on these pages is a benchmark somebody ran on hardware you do not have; where the honest answer is “measure it”, you get the command.

GGUF Filename Conventions on Hugging Face, Decoded

How to read architecture, size, fine-tune, quantization and shard index straight off a GGUF filename, and which parts of the name are convention rather than specification.

8 min read

Choosing Between Q4_K_M and Q4_K_S

What the M and S actually change inside a Q4_K model, derived from llama.cpp's per-tensor type assignment, and what the resulting size and quality gap is worth.

8 min read

What to Expect From Pure CPU-Only Local Inference

An upper bound on CPU generation speed derived from memory bandwidth and model file size alone, with the arithmetic shown and the command to measure the real figure.

9 min read

Why Your Local Model Feels Slower Than ChatGPT on Good Hardware

The gap is not hardware quality — it is that you are running a batch of one against a fleet that batches hundreds, and the arithmetic of that is decisive.

8 min read

Budgeting Disk Space for Keeping Several Local Models Around

A worked disk budget for a five-model library built from published file sizes, plus the duplicate copies three common tools create without telling you.

9 min read

Choosing a Quantization Level by Use Case

Why code and structured output degrade faster under quantization than open-ended chat, argued from what quantization does to the token distribution.

9 min read

Running Two Local Models Side by Side as a Fallback Pair

How to keep a fast primary and a stronger fallback resident at once, with the VRAM budget derived from weights plus KV cache for both.

9 min read

Verifying a Downloaded GGUF File's Checksum

Get the SHA256 that Hugging Face already publishes for a GGUF file, compute your own, and compare the two before the file is ever loaded.

8 min read

What Happens When You Quantize an Already-Quantized Model

Why llama-quantize refuses a quantized input by default, what --allow-requantize actually does to the weights, and what to do instead.

8 min read

Running a Local Model From a Portable USB Drive

A llamafile-based setup that runs entirely off a USB stick on an unmodified host, including the 4 GB executable ceiling and the filesystem choice that decides everything.

8 min read

Benchmarking Your Own Local Model's Tokens per Second

Produce a tokens-per-second figure for your own machine that you can defend, using the runtime's own timing fields rather than a stopwatch.

9 min read

Context Length or Quantization: Which Wins the Same VRAM Budget

Works out how many extra context tokens one quantization tier actually buys, from published parameter counts and attention geometry.

10 min read

Why the First Token Takes Longer Right After a Model Loads

Separates the four one-time costs paid on a first request: disk read, PCIe transfer, driver context creation and buffer allocation, and sizes each from published rates.

10 min read

GPU Power Draw While a Local Model Sits Idle but Loaded

Why a card with weights resident cannot drop to its lowest power state, what that costs per year, and how to measure your own figure.

9 min read

Thermal Throttling Running a Local Model on a Gaming Laptop

Reads the throttle reason flags nvidia-smi reports during a long generation and separates a thermal cap from a power cap, which have different fixes.

10 min read

Serving Concurrent Requests From One llama.cpp Server

How --parallel divides one KV cache into slots, the per-slot context arithmetic that decides how many fit, and what happens to the request that finds them all busy.

11 min read

Merging a LoRA or Keeping It Separate: Choosing for Local Deployment

The disk, memory and swap-flexibility consequences of baking an adapter into the weights versus loading it at runtime.

10 min read

The Difference Between Loading and Warming Up a Local Model

Why weight loading and the first forward pass are separate phases, what each one actually does, and why memory mapping moves work between them.

9 min read

What Running a Local LLM Actually Costs in Electricity

A per-hour and per-million-token electricity cost derived from stated wattage and a stated, dated tariff, with every input labelled.

10 min read

When a Local Model Silently Truncates Context You Thought It Kept

The log lines and response fields that reveal a context ceiling lower than you configured, and the four places the ceiling actually comes from.

10 min read

Why Two Quantization Tools Produce Different File Sizes for the Same Model

A quant name is a recipe rather than a byte layout, and the tensors most often given a different recipe are the two largest ones outside the layers.

10 min read

Accessing a Local Model Server From Another Device on Your Network

Binding a local model server to the network, with the authentication it does not have by default put in before the bind rather than after.

11 min read

Securing a Local Model Server Exposed to Your LAN

What changes the moment a local model server stops listening on loopback, and the three controls that put it back under your authority.

9 min read

A Local LLM Setup for Under $500: What the Budget Actually Buys

The arithmetic from a VRAM budget to a model size, and an honest method for pricing the hardware today rather than a number that was true last quarter.

10 min read

Why “Free” Local Inference Still Has a Real Cost

The zero-marginal-cost claim is a statement about variable cost only, and the fixed and idle costs it omits are usually the larger half.

9 min read

GPU Memory Not Freed After a Local Model Crashes

Why nvidia-smi can report gigabytes in use with no processes listed, and the order in which to reclaim it without rebooting.

9 min read

Running a Local Model as a systemd Service That Survives Reboot

A unit file that restarts a model server after a crash and comes back after a reboot, including the two defaults that quietly stop it doing either.

10 min read

When a GPU Supports Both INT8 and FP8, Which One Wins

Which tensor-core generation accelerates which 8-bit format natively, why their published peak rates are identical where both exist, and why that barely matters for single-stream decoding.

9 min read

A Local Model Won’t Stop Generating

Why a local server generates forever by default, how to tell an unrecognised stop token from a missing length cap, and the fix for each.

9 min read

Why a Local Model’s Output Isn’t Identical Run to Run at Temperature Zero

Greedy decoding is deterministic given identical logits; the logits are not identical, because the kernels choose their reduction order from the batch shape.

10 min read

Sampling Settings for a Local Coding Assistant

Why code generation wants a narrow distribution, derived from the cost of a single wrong token, and what each sampler parameter does to that distribution.

9 min read

What GPU Utilization Percent Actually Means During Local Inference

The metric is a duty cycle over kernel execution time, not a measure of how much of the GPU is working, which is why decode pins it at 100% while using a fraction of the hardware.

9 min read

Running a Local Model Behind a Reverse Proxy for Remote Access

A Caddy and an nginx configuration that front a local model server with TLS, including the buffering and timeout settings that silently break streaming.

10 min read

Other topics