Repeat Penalty Explained

The repetition penalty is not a rule about repeated phrases. It is an arithmetic operation on the logits of individual tokens that have appeared in a fixed-size window, and almost everything people find strange about it follows from that being all it is.

The formula

llama.cpp keeps a count of every token seen in the last penalty_last_n positions. For each candidate token in that set it applies three separate penalties, in one pass, in src/llama-sampler.cpp:

if (logit <= 0) { logit *= penalty_repeat; }
else            { logit /= penalty_repeat; }

logit -= count * penalty_freq + (count > 0) * penalty_present;

Three things are worth reading off that. The repeat penalty is multiplicative on the logit, so its effect scales with how large the logit already is. The frequency penalty is additive and proportional to how many times the token occurred. The presence penalty is additive and flat — it fires once, however often the token appeared. They are three different shapes of discouragement and they compose.

llama.cpp’s defaults are --repeat-penalty 1.00, which disables it, with --repeat-last-n 64, and both frequency and presence at 0.00. The disabled-by-default part is recent history worth noticing: the penalty is a blunt instrument and modern instruction-tuned models repeat far less than the models it was invented for.

The sign problem, and llama.cpp’s fix

The technique comes from the CTRL paper by Keskar and colleagues at Salesforce Research, September 2019 — arXiv:1909.05858 — which described penalised sampling as dividing the scores of already generated tokens by a penalty factor. Dividing is fine while the score is positive. It is exactly wrong when the score is negative: dividing −8 by 1.2 gives −6.67, which is higher, so the penalty would make an unlikely repeated token more likely.

The comment in llama.cpp’s implementation says so plainly, that the academic publication only divided, that this would make tokens with negative logits more likely, and that multiplying instead is the common fix. Hence the branch above.

The fix works, and it leaves an asymmetry you should know about. The amount subtracted from a token’s logit is proportional to that logit’s magnitude, in both branches. A penalty of 1.2 costs a token sitting at logit 12 about 2.0 logits, a token at 3 about 0.5, and a token at −8 gains a further 1.6 of suppression. So the same setting is a mild nudge for a marginal token and a serious demotion for the model’s favourite. It also means the setting does not transfer between models, because models differ in the scale of their logits before softmax; 1.15 on one can be as aggressive as 1.3 on another.

The window is a token count

--repeat-last-n defaults to 64, and the number is tokens, not words, sentences or turns. Sixty-four tokens is roughly a short paragraph of English and rather less of code or JSON, where the tokenizer spends ids on punctuation and indentation. Anything that scrolls out of that window stops being penalised entirely — which is why a long generation can loop at a period longer than the window and the penalty never notices.

Setting the window to 0 disables the penalty regardless of the multiplier. Setting it very large is not free either: every candidate token in the vocabulary is checked against the counts, and a wide window means more of the vocabulary is under penalty at once, which flattens the distribution generally rather than discouraging anything in particular.

Why a high value breaks factual repetition

The penalty cannot see structure. It sees token ids, and a great deal of correct text is the same id appearing again and again for reasons that have nothing to do with the model looping.

• Syntax. A JSON object is closing braces, commas and quote marks; a Python function is four-space indent tokens. Each of those is one id repeating dozens of times inside any 64-token window. Penalise them and the model produces structurally invalid output — which is the mechanism behind “my model stopped closing its brackets when I raised the repeat penalty”.
• Names. A document about one API repeats that API’s name. The penalty pushes the model towards a synonym, and for a proper noun the nearest thing to a synonym is a wrong name.
• Numbers and lists. Digit tokens recur inside a table. So does the list-item token. Penalising them corrupts the content rather than the style.

Work one case through. Suppose the model is 96% certain the next token is }, with the closing brace at logit 12 and its nearest rival at 8.5. A repeat penalty of 1.3 divides the brace down to 9.2 — still ahead. At 1.5 it falls to 8.0, and the rival now wins outright. Nothing about the model changed; a division did. This is the same arithmetic as the previous section, and it is why values above about 1.2 are risky on structured output and values above 1.3 are risky on anything.

What to reach for instead

If the goal is to stop verbatim looping rather than to discourage words, llama.cpp has samplers that target the loop specifically. DRY — --dry-multiplier with --dry-allowed-length, which defaults to 2 — penalises tokens that would extend an existing repeated sequence, so a brace that occurs a hundred times in unrelated places is untouched while a sentence starting to repeat itself is not. That is the behaviour most people wanted from the repeat penalty.

If the goal is variety of vocabulary, the additive frequency and presence penalties are better behaved than the multiplicative one, because their effect does not depend on the logit’s scale. And if the goal is to forbid one specific string, no penalty is the right tool: use a logit bias for a token or a grammar for a pattern. Loops that survive all of this are usually a prompt problem or a truncation-sampler problem — a very tight min-p or a near-zero temperature will produce loops that no penalty can reasonably undo.