Repetition Loops and Degenerate Output

The model writes a good paragraph, then a slightly worse one, then the same sentence four times, then that sentence forever until it hits your token limit. This has a precise mechanism, a published characterisation and a set of parameters that address it — and reaching for those parameters first is usually the wrong move.

What you are looking at

Degenerate repetition comes in several sizes, and the size tells you where to look. Token-level stutter — a word or short phrase repeated immediately — usually points at decoding parameters or at a quantised model. Sentence-level loops, where a full sentence recurs verbatim with small variations, are the classic case the literature studies. Paragraph-level cycling through two or three states typically appears in long generations. And structural repetition — the same list item, the same JSON key, the same section heading — is usually an under-specified schema rather than a sampling problem.

Where in the output it starts is the other free diagnostic. A loop that begins immediately usually means a malformed prompt or a chat template mismatch — the model is completing something other than what you think it is. A loop that begins after several hundred good tokens is the classic degeneration described below, and it is a property of the decoding rather than of the request. A loop that only appears on one model while others handle the same prompt cleanly points at that deployment’s quantisation, which flattens the distribution and makes near-ties between tokens more common.

Repetition amplifies itself

Holtzman, Buys, Du, Forbes and Choi’s The Curious Case of Neural Text Degeneration (ICLR 2020) is the paper that made this precise, and its central observation is the one to carry around: when a phrase repeats, the model’s probability for repeating it again increases with each repetition. The loop is not a plateau; it is a positive feedback system that gets harder to escape the longer it runs.

The reason is that the context is the model’s only evidence about what kind of document it is completing. Once the recent context contains “X. X. X.”, the most probable continuation of that document is X — the model is correctly predicting the continuation of a repetitive text, and it is repetitive because the model made it so. Xu, Liu, Lan, Gao and Li’s Learning to Break the Loop (2022) analysed this self-reinforcement effect directly and reported that it is stronger for sentences with higher initial repetition probability, which is why some phrasings lock in and others never do.

This is the same structural loop as language drift and as the self-conditioning on the long-conversation page: the model’s own output is the strongest evidence in its context about what it should do next. Repetition is that mechanism at its most visible.

Why maximisation decoding produces it

Holtzman et al.’s other contribution was the diagnosis. Decoding that maximises likelihood — greedy decoding, beam search — reliably produces degenerate repetition, while human text does not sit anywhere near the maximum-likelihood path. Human writing is full of moderately surprising word choices; the highest-probability continuation, chosen repeatedly, walks into a repetitive attractor and stays there.

Their proposed fix, nucleus sampling, is now the default nearly everywhere: sample from the smallest set of tokens whose cumulative probability exceeds p, so that the tail is truncated but the choice among plausible tokens stays stochastic. The practical consequence for you is worth stating directly — if you are seeing loops, check your temperature before you check anything else. Temperature 0 on a long free-form generation is the configuration this entire literature is about.

The penalties, defined

Three parameters are commonly available and they are frequently confused, including in vendor documentation. All three modify logits before the softmax.

Three cautions that matter more than the tuning. These penalties are blind to structure: a JSON generation with a repeated key, a table with a repeated column header, or code with a repeated identifier is punished for being correct, and a frequency penalty is a common undiagnosed cause of malformed structured output. They apply within a single response only, so they do nothing about a model repeating itself across turns. And they treat the symptom — if your prompt asks for twenty variations on something with five real variations, the loop is the model telling you the task is over.

Welleck et al.’s unlikelihood training (2020) is the training-time answer, penalising repetition in the loss rather than at decode time. Worth knowing exists; not available to an API caller.

A server-side loop detector

Because loops are self-reinforcing, the cheapest intervention is to stop the stream as soon as one is detectable rather than paying for the remaining tokens:

from collections import Counter

def loop_detected(text, n=8, threshold=3, window=1200):
    """True when any n-gram recurs threshold times in the recent window.
    n=8 words is long enough that legitimate prose rarely trips it;
    lower it for lists, raise it for code."""
    words = text[-window:].split()
    if len(words) < n * threshold:
        return False
    grams = Counter(tuple(words[i:i + n]) for i in range(len(words) - n + 1))
    return grams.most_common(1)[0][1] >= threshold

async def guarded_stream(prompt, max_tokens=2000):
    buf = ""
    async for chunk in stream_model(prompt, max_tokens=max_tokens):
        buf += chunk
        yield chunk
        if loop_detected(buf):
            # Abort. Retry once with a higher temperature and a nudge, or
            # return what you have with the loop truncated.
            raise LoopDetected(buf)

Tune n to your content, and log every trigger with the prompt that produced it. The distribution of triggering prompts is almost always informative: loops cluster on a specific task type, and the real fix is usually in that prompt rather than in a global parameter change.

Loops in structured output

Repetition inside a JSON array is a different bug with a different fix, and applying sampling penalties to it makes things worse.

• Constrain the schema. An array with maxItems and a uniqueItems constraint, enforced by the provider’s structured-output mode, removes the failure rather than discouraging it.
• Do not ask for a fixed count. “Give me ten examples” when six exist forces the model to pad, and padding is repetition. Ask for “up to ten” and let it stop.
• Deduplicate after parsing. Semantically, not by string equality — near-duplicate list items are the common form and an exact-match filter misses all of them.
• Watch for interaction with truncation. A loop that hits your token limit returns a truncated object, and you will spend the debugging session on the wrong end of the problem. Check the finish reason first.