Skip to content
Multigrid

Ollama and the desktop local-model runtimes

Ollama, LM Studio, KoboldCpp, Jan, LocalAI and the rest — what each one sets on your behalf, and what it does not tell you it set.

A desktop runtime is a set of defaults with a download manager attached. The inference is llama.cpp or something close to it; what you are actually choosing between is who decides your context length, how many layers go to the GPU, how long a model stays resident, and which of those decisions you are allowed to see. Most of the confusing behaviour people report — the model that answers instantly and then takes twelve seconds, the 128k model that forgets at 4,000 tokens, the machine that runs out of memory only under load — is a default doing exactly what it was configured to do, silently.

These pages take the defaults apart one at a time. Where a runtime publishes a number, it is quoted with its publisher; where it does not, the arithmetic is shown from inputs you can check, and the command that reads the real answer off your own machine is given instead of a figure somebody guessed. Nothing here was benchmarked, and nothing here needs to have been.

Ollama Modelfile Syntax, Line by Line

Every Modelfile instruction, what it writes into the built model, and a working file that sets parameters and a system prompt together.

9 min read

Building a Custom Model With ollama create

What ollama create does to a Modelfile, what each flag changes, and how to end up with a named local model you can run and share.

9 min read

Ollama's num_ctx Parameter and What It Actually Sets

num_ctx allocates a KV cache, not a promise about memory — here is the arithmetic, the default Ollama picks for you, and what moves when you change it.

9 min read

Ollama's keep_alive Setting and Model Unloading

The unload timer, every value keep_alive accepts, and how to reason about the memory you are renting against the load time you are avoiding.

8 min read

Ollama's REST API, End to End

The endpoints, the request and response fields that matter, and a script that talks to /api/generate and /api/chat with nothing but HTTP.

10 min read

Where Ollama Stores Models and How Much Disk They Use

The manifests-and-blobs layout, what each path holds on each platform, and how to work out real disk use when tags share weights.

9 min read

Importing a GGUF File Into Ollama

Take a GGUF you downloaded yourself, wrap it in a three-line Modelfile, and run it under Ollama with a template that matches the model.

9 min read

Ollama's Embeddings API, End to End

Generate vectors from a local embedding model over plain HTTP, with the batching, truncation and normalisation details that decide whether the vectors are usable.

9 min read

Choosing How Many Layers Ollama Offloads to GPU

What num_gpu counts, what Ollama does when you leave it alone, and how to derive an offload count from free VRAM instead of guessing.

9 min read

Ollama's num_parallel Setting for Concurrent Requests

How OLLAMA_NUM_PARALLEL turns one loaded model into several request slots, and exactly what that multiplies in memory.

8 min read

Setting a System Prompt in an Ollama Modelfile

Bake a system message into a model with SYSTEM, verify it survived the build, and understand why it sometimes appears to be ignored.

8 min read

Choosing a Quantization When You Pull a Model in Ollama

What the tag suffixes mean, what each one costs on disk, and how to work out bits per weight from sizes the registry publishes.

9 min read

Running Ollama on Windows via WSL2

Installing Ollama inside a WSL2 distribution and proving the GPU actually reached it, which is the step that fails without saying so.

9 min read

Running Ollama in Docker

A container serving the Ollama API with the GPU actually attached, plus where the model blobs live so you do not download them twice.

9 min read

The TEMPLATE Field in an Ollama Modelfile

What TEMPLATE renders, which variables it can see, and the specific failure modes a template that does not match the base model produces.

9 min read

Ollama's Default Context Length, and Why It Is Not the Number You Read

Three documented defaults, the precedence between them, and the silent truncation that happens when the one that wins is smaller than your prompt.

10 min read

Running Multiple Models at Once in Ollama

The two limits that decide how many models stay resident, the memory each one costs, and what the scheduler does when the budget runs out.

9 min read

Ollama's Environment Variables, Explained

What each OLLAMA_ variable changes at runtime, grouped by the problem it solves, and why setting one in your shell usually does nothing.

10 min read

Running LM Studio's Local Server in OpenAI-Compatible Mode

Starting LM Studio's HTTP server and calling it with the same code you would point at a hosted API, including where the compatibility stops.

9 min read

Which Quant to Download in LM Studio's Model Picker

How to read the quant list against your own free memory, with the bits-per-weight derived from published file sizes rather than guessed.

9 min read

LM Studio's GPU Offload Slider Explained

What the slider sets underneath, how to work out the right layer count, and why a nearly-full offload is much slower than a full one.

9 min read

Presets and System Prompts in LM Studio

What a preset bundles, what it deliberately excludes, and the places a saved preset changes behaviour you thought your request controlled.

8 min read

Loading a Model Straight From Hugging Face in LM Studio

Pulling a model from Hugging Face without leaving the app, including why a repository you can see in a browser sometimes does not appear in the search.

8 min read

LM Studio's Headless and CLI Mode

Running the server on a machine with no desktop session, using the daemon and the lms CLI rather than the graphical app.

9 min read

What Happens When You Exceed LM Studio's Context Window

Why LM Studio sometimes truncates silently and sometimes returns a 400, which setting decides, and what to change so it stops.

9 min read

Loading Two Models at Once in LM Studio

How LM Studio holds several models in memory at once, and the arithmetic that decides whether yours will fit.

10 min read

KoboldCpp, Start to First Response

Download one binary, point it at a GGUF, and have KoboldCpp's browser UI answering on localhost in a few minutes.

9 min read

GPT4All, Start to First Response

Installing GPT4All, downloading a model and holding a fully offline conversation with no account, plus an honest note on the project's release cadence.

9 min read

Jan, Start to First Response

Installing Jan, pulling a model from its Hub, chatting locally, and turning on the OpenAI-compatible server on port 1337.

9 min read

LocalAI, Start to First Request

Getting a self-hosted, OpenAI-compatible endpoint answering a curl request, and understanding the model YAML that decides how it behaves.

10 min read

text-generation-webui, Start to First Response

Getting a model loaded and answering in oobabooga's web UI, now called TextGen, including the extension trap that catches most first attempts.

10 min read

llamafile Explained: A Model and Runtime in One Executable

How Cosmopolitan Libc lets one file execute unmodified on six operating systems and two instruction sets, with the weights inside it.

9 min read

Running a Model With llamafile, No Install Required

Download one executable, make it executable, and get a chat response — plus the three platform problems that stop it.

9 min read

KoboldCpp's Context Shifting Explained

Why moving the KV cache instead of reprocessing the prompt removes almost all of the pause between turns, and what silently turns it off.

9 min read

Connecting SillyTavern to a Local Backend

Getting SillyTavern's front end talking to a llama.cpp or KoboldCpp server on your own machine, and choosing the right API mode.

10 min read

Open WebUI as a Self-Hosted Chat Front End

Running Open WebUI in Docker, pointing it at your own model server, and setting up accounts before anyone else can reach it.

10 min read

Other topics