Connecting SillyTavern to a Local Backend

SillyTavern generates no text. It is a front end that assembles a prompt from characters, personas, world info and history, and sends it to something else. Connecting it to a local backend is therefore two separate jobs: get a server running, then tell SillyTavern what shape of API it speaks.

Start the backend first

Nothing in SillyTavern works until a server is answering, so bring one up and confirm it in a browser before you touch the connection panel. The two usual choices publish on different ports:

• KoboldCpp listens on 5001. It serves its own Kobold API under /api and an OpenAI-compatible surface under /v1, both from the same process. Getting it running takes about five minutes.
• llama.cpp’s llama-server defaults to 127.0.0.1 on port 8080, per its README, and serves an OpenAI-compatible API plus its own endpoints. Note that its context size defaults to 0, meaning “whatever the model declares”, which on a large model can be more than your machine can hold — set -c explicitly.

# llama.cpp
llama-server -m ~/models/qwen3-8b-q4_k_m.gguf -c 16384 -ngl 999 --port 8080

# KoboldCpp
./koboldcpp --model ~/models/qwen3-8b-q4_k_m.gguf --usecublas \
            --gpulayers -1 --contextsize 16384 --port 5001

Open the server’s own page in a browser and get one reply out of it. If that fails, the problem is not SillyTavern and no amount of fiddling with the connection panel will help.

Other backends work too — TextGen exposes an OpenAI-compatible API, and so do LM Studio and Ollama — but the two above are the ones with dedicated API types in SillyTavern, which means their native sampler parameters are exposed rather than flattened into the OpenAI subset. If a sampler you want is missing from the UI, that is usually the reason: you connected through a generic path rather than the specific one.

Text Completion versus Chat Completion

SillyTavern’s API dropdown separates these, and the choice changes what your backend receives. It is the single decision that most affects output quality with a local model.

Chat Completion sends a structured array of messages with roles — system, user, assistant — and the backend applies its own template to turn that into a prompt string. Formatting is the server’s job.

Text Completion sends one string and asks the model to continue it. SillyTavern builds that string itself, from your character card, persona, world info and history, using the instruct template you selected.

For local roleplay backends, Text Completion is generally the mode you want, and the reason is control rather than preference. Everything SillyTavern is good at — character cards, author’s notes, world info injected at specific depths, prefills that force the model to continue in a particular voice — requires putting text at exact positions in the prompt. A chat-completions endpoint reserves that arrangement for itself, so those features either degrade or stop working. Chat Completion is the right choice when you are pointing SillyTavern at a hosted API that only offers that shape.

Making the connection

1. Open the API connections panel — the plug icon in the top bar.
2. Set API to Text Completion.
3. Set the API Type to match your server: KoboldCpp for KoboldCpp, llama.cpp for llama-server. These are separate entries because their native endpoints and their sampler parameter names differ; picking the wrong one connects and then behaves oddly.
4. Enter the URL: http://127.0.0.1:5001 for KoboldCpp, http://127.0.0.1:8080 for llama.cpp. Use 127.0.0.1 rather than localhost if the connection hangs — on some systems the name resolves to IPv6 first and the server is only bound to IPv4.
5. Press Connect. A successful connection reports the loaded model name. If it reports a model, the pipe is open.

Templates decide the output quality

A connection that works and produces bad output is almost always a template mismatch, and this is where most of the disappointment with local models comes from. In Text Completion mode SillyTavern is building the prompt, so it needs to know the exact instruct format the model was trained on — which special tokens delimit a turn, how the system message is marked, what the model expects to see immediately before its own reply.

Choose the instruct template that matches your model’s documented chat format, and set the context template to match. Two symptoms map straight back here: a model that writes both sides of the conversation has a missing or wrong stop sequence, and a model that ignores its character card has a system block the template put somewhere the model does not read as instructions. Set the context size in SillyTavern to the same value you gave the server as well — if the front end thinks it has 32k and the server was started with 16k, the server truncates and you never see it happen. What happens once you reach that limit on KoboldCpp is worth knowing before it does — Context Shifting keeps the conversation moving by discarding the oldest tokens, and editing an old message is the thing that turns it off.

Reaching it from another device

SillyTavern itself serves on port 8000 and its config.yaml reference documents the relevant defaults: listen is false, whitelistMode is true, and basicAuthMode is false. In other words, out of the box it is loopback-only with IP filtering on and no password — safe because it is not reachable, not because it is protected.

Setting listen: true to reach it from a tablet changes that in one step. The documentation is direct that disabling security measures is strongly discouraged, and the reason is concrete: with listen on and no authentication, anyone on the network gets your chats and your backend. If you turn it on, leave the whitelist enabled, turn on basicAuthMode with real credentials, and remember that your inference server is a second unauthenticated listener that needs the same treatment. A reverse proxy or an SSH tunnel is a better answer than either.