KoboldCpp, Start to First Response

KoboldCpp is llama.cpp packaged as a single self-contained executable with a web UI attached. There is no install step, no Python environment, and no package manager: you download one file, hand it a GGUF, and it serves a chat interface on port 5001.

Getting the right binary

Builds are published per release on LostRuins/koboldcpp’s GitHub releases page — v1.118.1, dated 1 August 2026, is the current one at the time of writing. Pick by what your machine has, not by what sounds fastest:

• Windows, NVIDIA — the full koboldcpp.exe, which bundles the CUDA libraries. It is large for exactly that reason.
• Windows, AMD or Intel — the same executable, launched with Vulkan rather than CUDA. Vulkan works across vendors; CUDA is NVIDIA only.
• Linux — the koboldcpp-linux-x64 binary. Mark it executable with chmod +x before running it.
• macOS — the macOS build, which uses Metal on Apple Silicon. There is no separate flag to enable it.

The binary is genuinely self-contained: it carries the inference engine, the GPU backends and the web front end inside one file, so there is nothing to uninstall later and nothing on the system path. That also explains the download size, which is dominated by the GPU libraries rather than by any model — no weights ship with it. If the size bothers you, the no-CUDA builds are much smaller and run on Vulkan or CPU.

Getting a GGUF

KoboldCpp loads GGUF files and only GGUF files. Any of the well-maintained quantisation repositories on Hugging Face will do; what matters is picking a size that fits. A rough rule that you can check rather than trust: at Q4_K_M a GGUF weighs roughly 0.61 bytes per parameter, so an 8B model is about 5 GB and a 13B about 8 GB — the derivation of that per-parameter figure is on the combined memory budget page, and the trade-off between quantisation levels is in choosing a quantisation. Put the file somewhere with a short path; the launcher takes the path as a string and a space in a directory name is the most common first-run mistake.

Launching it

1. Run the executable with no arguments. KoboldCpp opens a launcher GUI with the model picker and the acceleration options. This is the easiest first run and it is not a lesser mode — every setting in it has a command-line equivalent printed when you start.
2. Browse to your GGUF, choose CUDA for NVIDIA or Vulkan otherwise, and leave the layer count on its automatic setting for now.
3. Press Launch. The console prints the load, the layer offload it decided on, and finally a line telling you to connect to the local endpoint.
4. Once you know what works, skip the GUI entirely and run it from a shell so the settings are recorded somewhere:

# NVIDIA, 16k context, let it decide the layer split
./koboldcpp --model ~/models/qwen3-8b-q4_k_m.gguf \
            --usecublas --gpulayers -1 --contextsize 16384 \
            --host 127.0.0.1 --port 5001

# AMD, Intel, or anything without CUDA
./koboldcpp --model ~/models/qwen3-8b-q4_k_m.gguf \
            --usevulkan --gpulayers -1 --contextsize 16384

Passing flags suppresses the launcher. If you want both — flags as defaults, GUI to adjust them — add --launch, which the project’s wiki documents as showing the GUI even when command-line flags are present.

First response in Kobold Lite

Open http://localhost:5001. The bundled front end is Kobold Lite, a full chat, story and instruct interface served by the same process — nothing else to install. Type a message and send it. The first response is slower than the rest, because the prompt has to be processed before any token comes back; subsequent turns in the same conversation reuse what has already been processed.

If the response is coherent, you are done: a locally loaded GGUF is answering in a browser. If it is gibberish or ignores your instruction format, the model is loaded but the prompt template is wrong — switch Kobold Lite to Instruct mode and choose the template family that matches your model rather than assuming the default fits.

Read the console rather than only the browser. The startup output tells you the architecture it detected, how many layers went to the GPU against how many the model has, and the context size it settled on. Those three lines answer most of the questions people ask afterwards. A load that stops with a complaint about an unknown or unsupported architecture is not a broken download — it is a GGUF newer than the llama.cpp inside your build, and the fix is a newer KoboldCpp release rather than a different quantisation of the same model.

The flags you will change next

• --gpulayers — how many transformer layers go to the GPU. Passing -1 asks KoboldCpp to guess, which the wiki is candid about being “often not the most accurate”. Recent versions also offer --autofit, which sizes the split against available memory. If generation is far slower than you expect, this is almost always the flag: layers that did not fit are running on CPU.
• --contextsize — the window. Raising it raises KV cache memory linearly, so it competes with --gpulayers for the same VRAM.
• --noshift — disables Context Shifting, which is on by default and is why long conversations do not stall on reprocessing. Leave it alone unless you have a reason; what it actually does is worth reading before you turn it off.
• --host — defaults to loopback. Changing it to 0.0.0.0 puts an unauthenticated model server on your network. Do that deliberately or not at all.

The same process also serves two APIs on 5001: KoboldCpp’s native endpoints under /api, and an OpenAI-compatible surface under /v1. That second one is what lets an existing OpenAI client talk to it with only a base-URL change, and it is how SillyTavern connects to a local backend.