Sampling Settings for a Local Coding Assistant

The usual advice is “use a low temperature for code”, which is correct and explains nothing. The reason is a property of code rather than of models, and once you have it the rest of the sampler settings follow without needing to be memorised.

The cost of one wrong token in code

At most positions in a piece of prose, several continuations are acceptable. “The function returns a” can be followed by dozens of words that all produce a sensible sentence, and choosing an unlikely one costs a little style. Sampling from the tail is cheap because the tail contains usable options.

Code is not like that. At most positions the grammar admits one or two tokens: the closing bracket that matches the one you opened, the identifier that was declared four lines up, the keyword that the language requires next. The set of acceptable continuations is small and often has exactly one member, so the tail of the distribution is not a set of alternatives — it is a set of errors.

The second half of the argument is the one that makes it decisive. Generation is autoregressive: each token is appended to the context and conditions everything after it. A model that emits a wrong identifier does not notice and correct it on the next token; it conditions on the wrong identifier and continues consistently with the mistake, because a consistent continuation is what is most likely given the text so far. One sampled error therefore does not produce one wrong token, it produces a wrong function. In prose, a low-probability word is a stylistic wobble the next sentence recovers from; in code, it is a variable name that does not exist, repeated three more times because it now looks like it does.

So the derivation is: narrow set of correct continuations, plus amplification instead of correction, equals a distribution you want truncated hard.

What lowering temperature actually does

Temperature divides the logits before the softmax:

p_i  =  exp(z_i / T)  /  sum_j exp(z_j / T)

T = 1     leaves the model's own distribution unchanged
T > 1     shrinks the gaps between logits -> flatter, more diverse
T < 1     magnifies the gaps            -> peakier, more confident
T -> 0    all mass moves to the argmax  -> greedy decoding

The important part is that temperature does not add information or improve judgement. It is a monotone transform: it never reorders the tokens, only redistributes probability mass between them. Lowering it makes the model more likely to pick what it already thought was most likely. If the model’s top choice is wrong, temperature 0 gives you that wrong answer reliably rather than a right one.

That is the honest limit of the advice. Low temperature buys you consistency and removes tail errors; it does not buy correctness. It also does not buy determinism — a common assumption, and why temperature-zero output still varies run to run explains why greedy decoding and reproducible output are separate things.

Vendors publish task-dependent recommendations that agree with this reasoning. DeepSeek’s API documentation includes a table of recommended temperatures by use case and gives 0.0 for coding and mathematics, against materially higher values for conversation and creative writing — see DeepSeek’s parameter settings page. Model cards for code-specialised open models sometimes recommend a non-zero value instead, on the grounds that a little diversity helps when you intend to generate several candidates and pick one. Both positions are consistent with the mechanism above; they differ on whether you are producing one answer or a pool.

top-k, top-p and min-p do different jobs

These three all remove tokens before sampling, and they differ in what they use to decide.

• top-k keeps a fixed number of the highest-scoring tokens. It does not care whether the model is confident: it keeps 40 candidates when there is exactly one correct closing brace, and it keeps 40 when the model is genuinely uncertain. llama.cpp documents the default as 40, with 0 meaning disabled.
• top-p (nucleus) keeps the smallest set of tokens whose probabilities sum to p. This adapts: where the model is confident, the nucleus is one or two tokens; where it is unsure, the nucleus widens to admit more. Documented default 0.95, with 1.0 meaning disabled.
• min-p keeps tokens whose probability is at least a given fraction of the top token’s probability. Documented default 0.05, meaning “anything at least a twentieth as likely as the leader”. This is the most directly aimed at the problem described above, because it prunes relative to the model’s own confidence rather than to an absolute count or an absolute mass.

The interaction that confuses people: at temperature 0, all three are irrelevant. The argmax token survives every truncation filter by construction, so setting top_p to 0.8 alongside temperature: 0 changes nothing at all. The truncation parameters only earn their keep when you have deliberately kept a non-zero temperature — and that is the case where a low top-p matters, because the tokens temperature has left alive at the bottom of the distribution are exactly the syntactically invalid ones.

Why repetition penalties hurt code

Repetition penalties reduce the logit of any token that appeared in the recent window — llama.cpp documents --repeat-last-n with a default window of 64 tokens and --repeat-penalty with a default of 1.00, which is disabled. The default is disabled for good reason and many front-ends override it to 1.1 anyway.

In prose, penalising recently seen tokens discourages the model from looping. In code, the tokens that recur within any 64-token window are the ones the language requires: indentation, the closing brace, the semicolon, self, the loop variable, the name of the object you are calling four methods on. Penalising them makes the correct token less likely precisely where it is most obviously correct, and because penalties are applied to the logits before temperature, they can flip the argmax even under greedy decoding. The failure looks like a model that renames variables mid-function or drops a brace, and it is reliably blamed on the model.

Genuine degenerate repetition — the same three lines forever — is a real failure mode with a different remedy: raise temperature slightly, or find the prompt structure that induced it. Reaching for the penalty is treating a symptom with a tool that damages the thing you wanted. If output is instead running on past the end of the answer entirely, that is a stopping problem, covered in when a local model will not stop generating.

A defensible starting point

# One answer you intend to use as-is (refactor, fix, completion):
{
  "temperature": 0,
  "top_k": 1,
  "repeat_penalty": 1.0,
  "max_tokens": 1024
}

# Several candidates you intend to filter by running them:
{
  "temperature": 0.2,
  "top_p": 0.8,
  "min_p": 0.05,
  "top_k": 0,
  "repeat_penalty": 1.0,
  "max_tokens": 1024
}

The first block is greedy and states it twice, deliberately: top_k: 1 forces argmax even in a stack that clamps temperature rather than switching to greedy. The second keeps just enough temperature to produce distinct candidates and then truncates the tail hard with a low top-p, which is the configuration the derivation argues for whenever the temperature is not zero.

Two things worth more than any of these numbers. The quantisation level matters more for code than for prose, because the same tail sensitivity that makes sampling risky makes precision loss visible — quantisation level by use case treats that. And a compiler or a test suite is worth more than any sampler setting: the settings above reduce the rate of invalid output, while running the code tells you which output was invalid.