Ollama in a Development Workflow

Ollama is a model manager and a local HTTP server around an inference engine. Its value in a development workflow is not that it runs models — several things do — but that it makes a model a named, pullable, versioned dependency and puts it behind an endpoint your application can treat exactly like a hosted provider.

What it adds over llama.cpp

Underneath it uses llama.cpp for inference, so the performance characteristics are the ones derived on that page — bandwidth-bound generation, quantisation controlling model size, layer offload controlling where the work happens. What sits on top is the part worth having.

• Model management. Pull by name, store once, share across projects, remove when disk runs short. No manual file handling and no wondering which GGUF a project expects.
• A resident server. One background process serves every project on the machine, loading and unloading models on demand rather than paying startup cost per invocation.
• Sensible defaults per model. The chat template, stop tokens and default parameters travel with the model, so you are not reconstructing prompt formats by hand.
• An OpenAI-compatible endpoint. Which is what makes it a drop-in for development against code written for a hosted API.

The commands you will actually use

ollama pull  <model>      # download; models are tagged, e.g. name:8b
ollama run   <model>      # interactive chat, pulling first if needed
ollama list               # what is on disk, with sizes
ollama ps                 # what is loaded in memory right now
ollama show  <model>      # template, parameters, licence, model details
ollama rm    <model>      # reclaim the disk

ollama show is the underused one. It prints the chat template and the default parameters the model runs with, which is where you find out that a model has a system prompt baked in or a stop sequence that explains truncated output. When a local model behaves oddly, read this before changing your prompt.

ollama ps matters on shared or memory-constrained machines, because loaded models occupy memory for a keep-alive window after their last use. Two large models pulled during an afternoon of experimenting can leave a machine swapping for reasons that are not obvious from any application log.

Modelfiles: a model plus your settings

A Modelfile is a small declarative file that derives a new named model from an existing one with your system prompt and parameters attached. It is Docker-flavoured on purpose, and it is the right place to put settings that must be identical for everyone on the team.

FROM <base-model>:<tag>

SYSTEM """You answer questions about the billing system.
Answer only from the context provided. If it is not there, say so."""

PARAMETER temperature 0.2
PARAMETER num_ctx 8192

ollama create billing-assistant -f ./Modelfile
ollama run billing-assistant

Two honest limits. A Modelfile is a local artefact: creating it on your machine does not put it on your colleague’s, so the file belongs in the repository with a command in the README or a make target that builds it. And a system prompt baked into a model is invisible to application code, which is a real hazard — someone debugging your application will not think to run ollama show. For anything that affects behaviour in a way a reader of the code should see, keep the prompt in the application and use the Modelfile only for parameters.

Two APIs, and which to target

Target the compatible endpoint from application code. The reason is not preference: it means the same code runs against a local model in development and a hosted provider in production, with one environment variable different. Use the native endpoint from scripts and tooling where you want an option the compatibility layer does not expose.

Wiring it into your application

The seam is a base URL and a model name in configuration. If those two are hardcoded anywhere, this does not work; if they are in environment, the switch is free.

# development
OPENAI_BASE_URL=http://localhost:11434/v1
OPENAI_API_KEY=ollama            # required to be non-empty, not checked
MODEL=<local-model>:<tag>

# production
OPENAI_BASE_URL=https://<your gateway or provider>/v1
OPENAI_API_KEY=<real key>
MODEL=<hosted-model-id>

For continuous integration, running Ollama as a service container and pulling a small model in a setup step gives tests a real model without network access to a provider or a spend line — the pattern discussed in testing without the model for the cases where a stub is not enough. Keep the model small; the pull dominates the job time.

Choosing which model to pull

The catalogue is large and the tags are cryptic, but the sizing question is arithmetic rather than taste. A model runs comfortably when its file plus its cache fits in the memory that will hold it — video memory if you want speed, system memory if you are prepared to wait.

file size ≈ parameters × bits_per_weight / 8

  8B at ~4.5 bits (a typical default 4-bit tag)  ≈ 4.5 GB
  8B at 8 bits                                   ≈ 8.0 GB
  70B at ~4.5 bits                               ≈ 40 GB

then add:
  KV cache  = 2 × layers × kv_heads × head_dim × bytes × context_tokens
  runtime overhead ≈ 0.5–1 GB

Rule of thumb for a machine with V GB of usable video memory:
  choose a tag whose file size is below about V − 2 GB.

Two decisions follow from that. On a 16 GB machine, an 8B model at a 4-bit tag leaves room for a real context; the same model at 8 bits does not, and the quality difference between them is far smaller than the difference between running on GPU and spilling to CPU. And on Apple silicon, where memory is unified, the constraint is total system memory shared with everything else, so leave more headroom than the arithmetic suggests.

For which model rather than which size, the ordinary evaluation discipline applies: pick two candidates, run your own twenty hardest inputs through both, and read the outputs. Local model leaderboards move constantly and none of them are measuring your task — the argument is in local model quality.

The four things that surprise people

The default context length is short. Shorter than most models support, and input beyond it is silently truncated rather than rejected. The symptom is a model that ignores the beginning of a long prompt — the retrieved documents, typically — with no error anywhere. Set the context length explicitly, in the Modelfile or per request via the native API’s options, and check what your version defaults to rather than assuming it matches the model card. This single default accounts for a large share of “the local model is much worse” reports.

The default tag is not a specification. Pulling a name without a tag gets whichever variant is the current default, which is typically a mid-size, mid-quantisation build and can change. Pin the full tag in anything shared, or two developers will be running different models and comparing results.

Concurrency is limited by default. It is built for a developer machine, not for serving. Parallel request handling and the number of simultaneously loaded models are controlled by environment variables, and the defaults are conservative. If a load test against Ollama is slow, that is not a statement about the model. For serving, use vLLM or TGI.

Binding to all interfaces exposes an unauthenticated API. Changing the host variable to serve other machines publishes an endpoint with no authentication that can load and run models. On a laptop on a shared network, that is an open service. Bind to localhost, or put it behind something that authenticates.