Skip to content
Multigrid

Open-weight models & local inference

What you can actually run on hardware you control, what the licence actually lets you do with it, and how to work both out from first principles rather than from folklore.

Writing about local inference tends to be either a shopping list or a manifesto. Neither helps with the two questions people actually arrive with: will this model fit on the hardware I have, and am I allowed to build a business on it.

Both have real answers. The first is arithmetic — parameters, bits per weight, layers, key-value heads and context length, and there is no guesswork in it once you have those five numbers. The second is a licence, and the licences differ far more sharply than the phrase “open source model” suggests. These pages do the arithmetic in the open and name the licence families precisely, then stop short of telling you which model is best this month, because that changes faster than a page can and the selection procedure is the durable part.

Open Weights vs Open Source: The Distinction That Matters

What each licence family in the open-weight world actually permits regarding commercial use, redistribution and derivatives — and why most of them are not open source.

5 min read

Running an LLM on Your Own Laptop: A Complete Guide

The memory arithmetic that decides what will fit, a worked example, and the commands to get a quantised model answering on your own machine.

6 min read

Ollama, llama.cpp and LM Studio Compared

What each of the three actually is, which decisions each one takes away from you, and a harness for running your own fair comparison.

5 min read

GGUF, Safetensors and Model File Formats

What each model file format guarantees about loading, quantisation and metadata — including the one that can execute code on your machine.

5 min read

Which Quantisation Should You Download?

What each quantisation level costs you in memory, calculated rather than guessed, and an acceptance test that tells you whether it costs you anything in quality.

6 min read

Local Model Quality: How Close to the Frontier?

Why the honest answer is task-shaped rather than a single gap, how to read a leaderboard without being misled, and how to build the only evaluation that decides your case.

5 min read

When Local Inference Is Cheaper Than an API

The full cost model for self-hosted inference, worked through to a cost per million tokens you can compare directly against a published price.

5 min read

Privacy as the Real Case for Local Models

The workloads where running your own model wins on grounds other than cost, and an honest account of what local inference does not protect you from.

5 min read

Serving an Open Model to Your Whole Team

How to size one machine for ten concurrent users, with the memory and throughput model written out and the load test that checks it.

5 min read

Fine-Tuning Open Models: The Full Pipeline

Data, LoRA, evaluation and serving, with the memory arithmetic that decides which of them you can afford and the failure points at each stage.

6 min read

Open Model Families: Who Ships What

The labs that publish open weights, the size classes they publish in, the licence character to expect from each, and how to refresh the picture yourself.

5 min read

Small Language Models: What 1–8B Can Actually Do

The tasks where a model in the one-to-eight-billion range is genuinely sufficient, the tasks where it is not, and the techniques that move the boundary.

5 min read

Chinese Open Models and Why They Matter

Why a large share of open-weight releases now come from Chinese labs, what is structurally distinctive about them, and how to evaluate one without guessing.

5 min read

Uncensored and Abliterated Models: What Changes

What the various modification techniques actually do to a model, what they cost beyond the refusals, and how to test whether one is fit for your use.

5 min read

Embedding Models You Can Self-Host

Why embeddings are the easiest thing to self-host, the storage and throughput arithmetic, and how to choose a model on your own corpus rather than on a leaderboard.

5 min read

Building an Offline AI Workstation

How to derive a hardware specification from the models you intend to run, which constraints actually bind, and what to check before buying anything.

5 min read

Hybrid Architecture: Local for Most, API for Hard Requests

A routing rule for sending most traffic to a model you control and the rest to a frontier API, with the cost model that says whether it was worth it.

5 min read

Prompt Portability From Frontier to Open Models

What breaks when a prompt written for a frontier model is pointed at a smaller open one, why each thing breaks, and the repair for each.

5 min read

Community Fine-Tunes: Are They Worth Using?

How to evaluate a community fine-tune against its base honestly, why the reported scores so often fail to reproduce, and what to check before deploying one.

5 min read

The Gap Between Open and Closed, Measured Over Time

How to define and track the capability lag between open-weight and closed models honestly, and the four distortions that make most published versions of the chart wrong.

5 min read

Other topics

Open-weight models & local inference · Multigrid