Logit Bias for Local Inference in llama.cpp

A logit bias is the bluntest control llama.cpp offers and the only one that is exact: it adds a number of your choosing to one token’s score, every step, before anything else in the sampler runs.

What the flag does to a logit

At each step the model produces one score per vocabulary entry. The logit-bias sampler walks your list of pairs and adds the bias to the matching entry’s score. That is the entire operation — there is no scaling, no decay, no dependence on what has been generated so far. The token is biased identically at position 1 and position 900.

Where it sits matters. In llama.cpp’s common/sampling.cpp the logit-bias sampler is added to the chain before the configured sampler list, so it runs ahead of penalties, before every truncation sampler, and before temperature. Two consequences follow. A positive bias can rescue a token that top-k or min-p would otherwise have cut, because the cut happens afterwards on the biased score. And temperature, being last, divides your bias along with everything else: a bias of b applied before a temperature of T is a bias of b/T in the distribution you finally sample from. At the default temperature of 0.8 your bias is quietly amplified by a quarter; at 2.0 it is halved.

The unit of a bias

Because probabilities are exp(logit) / Z, adding b to one logit multiplies that token’s unnormalised weight by exp(b) while leaving every other token’s alone. The bias is therefore a log-odds adjustment, and the conversion is worth memorising:

• b = +0.7 — roughly doubles the token’s odds against the field.
• b = +2.3 — roughly ten times the odds.
• b = -2.3 — roughly a tenth.
• b = -10 — about a 22,000-fold reduction, which for a token the model was not strongly committed to is indistinguishable from a ban.

Concretely: a token at 5% probability with everything else fixed, given b = +2.3, has its odds go from about 0.053 to 0.53, which is a probability near 35%. The odds move by a clean factor; the probability does not, because the rest of the distribution has to make room. If you reason in probabilities you will keep being surprised by biases that “did nothing” on a token the model had already written off.

The syntax, exactly

On the command line the argument is -l or --logit-bias and its format is TOKEN_ID(+/-)BIAS — a token id, then a literal plus or minus, then the magnitude, with no spaces. The parser reads the id, then the sign character, then the number, and throws invalid input format if any part is missing. llama.cpp’s own help text gives --logit-bias 15043+1 to increase the likelihood of the token ‘ Hello’ and --logit-bias 15043-1 to decrease it.

To find the id, tokenize the string with the same model — the ids are properties of the model’s vocabulary and are not portable between models, not even between quantisations of different families:

llama-tokenize -m models/target.gguf -p " Hello" --ids

llama-completion -m models/target.gguf \
  -p "Say hello." --logit-bias 15043-3

Note the leading space in the string being tokenized. Most tokenizers treat “Hello” at the start of a line and “ Hello” mid-sentence as two different ids, and biasing one of them is a common reason a bias appears to have no effect.

Over the server’s HTTP API the same control is the logit_bias request field, which takes a list of pairs — the documented example is "logit_bias": [[15043,1.0]] — and also accepts strings in place of ids, in which case every token of the string is biased individually. An OpenAI-style object mapping is accepted too, for compatibility.

Banning a token outright

A large negative number is a heuristic; the server gives you the real thing. Passing false as the bias value bans the token, and the documentation states the guarantee directly: "logit_bias": [[15043,false]] ensures the token is never produced.

curl -s http://127.0.0.1:8080/completion \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"Greet the user in one word:",
       "n_predict":8,
       "logit_bias":[[15043,false]]}'

Use that form rather than inventing a very negative float. A hard ban is unambiguous, survives a temperature change, and does not depend on the model’s logit scale — all three of which a -100 does not give you.

What logit bias cannot do

It operates on single tokens with no memory, and every limitation is a restatement of that.

• It cannot ban a word. A word is usually several tokens, and often several different tokenizations depending on leading whitespace and case. Banning one id leaves the model free to spell the same word another way.
• It cannot express a rule. “Never say this inside a code block” is a conditional, and the sampler has no notion of where it is. For anything conditional you want a grammar, which is a state machine over the output — see GBNF grammars.
• It has collateral damage. Banning a common subword removes it from every word that contains it. The model will route around the hole, sometimes by producing a misspelling, because the constraint is on the id and not on the meaning.
• It does not shrink the vocabulary. Biasing a thousand tokens costs a lookup per biased token per step; the model still computes every logit. This is a control, not an optimisation.

The one place a single-token bias is exactly the right instrument is the end-of-sequence token, because length really is a property of one id. A negative bias on EOS makes the model keep going and a positive one makes it wrap up sooner, and both are smoother than a hard token cap, which truncates mid-sentence. The server also exposes ignore_eos as a boolean for the extreme version of the same idea; prefer the bias when you want a nudge and the boolean when you genuinely want the stop suppressed. Find the id from the model’s own metadata rather than guessing — llama.cpp prints the EOS token when it loads the model, and it differs between base and instruction-tuned releases of the same family.

One last piece of behaviour that surprises people: the bias applies to drafted tokens too. Under speculative decoding the target model’s sampler is what verifies each draft, so a token you banned cannot be smuggled in by the draft model — but a heavily biased token also lowers the acceptance rate, because the draft model does not share your bias and keeps proposing what you are suppressing.