Migrating Logit Bias Usage Between Providers

logit_bias is one of the least portable parameters in any model API. It is keyed by token ids from a specific tokenizer, so a map tuned against one model is meaningless against another — and most target providers do not accept the parameter at all.

What logit_bias actually does

OpenAI’s Chat Completions API accepts logit_bias as a JSON object mapping token ids, expressed as string keys, to bias values between -100 and 100. The bias is added to the model’s logit for that token before sampling. Small values nudge; the documentation describes values around -1 to 1 as decreasing or increasing likelihood of selection, and values at the extremes as approaching a ban or an exclusive selection.

# "50256" is a token id, not a word. -100 ≈ ban.
resp = client.chat.completions.create(
    model=MODEL,
    messages=msgs,
    logit_bias={"1904": -100, "9642": 5},
)

Two properties follow directly and both are the source of the trouble. The bias applies to a token, not to a word, a phrase or a concept. And it is applied at every generation step, not once, so a token you biased down is biased down in every position where it might appear — including inside unrelated words that happen to share it.

This is why using logit_bias as a content filter reliably disappoints. A word you want to suppress is usually several tokens; you must bias every token that could begin it, and every variant with a leading space, capitalisation or punctuation. Suppress the leading token and the model frequently reaches the same word by a different tokenisation, because the distribution over alternative segmentations is not something the bias touched. You have made the word rarer, not impossible, and you cannot tell which from the response.

Why the map is not portable

The keys are ids in a particular vocabulary. Recent OpenAI models use a different encoding than older ones — the change from the cl100k_base generation to o200k_base renumbered the vocabulary — so a bias map built for one is silently wrong for the other. Wrong in the worst way: the ids still exist, so no error is raised. They now point at different tokens, and you have applied a -100 bias to something arbitrary.

That means a logit_bias map is not a configuration value. It is a derived artefact of one tokenizer, and it must be regenerated whenever the model changes, not copied. The only safe way to store one is as the source strings plus the encoding name, with the ids computed at load time:

import tiktoken

BANNED = ["Sorry", " Sorry", "sorry", " sorry"]

def bias_map(model: str, value: int = -100) -> dict[str, int]:
    enc = tiktoken.encoding_for_model(model)
    ids = {i for s in BANNED for i in enc.encode(s)}
    return {str(i): value for i in ids}

Even that is only correct within one vendor’s tokenizer family. Across vendors there is no correspondence at all: the same string splits into a different number of tokens with different ids, which is the same mechanism the library describes in the tokenizer comparison and in the tokenizer mismatch bug. A migration cannot translate a bias map. It can only rebuild one, and only if the target accepts the parameter.

When the target has no equivalent

Frequently it does not. Anthropic’s Messages API does not document a logit bias parameter; Google’s Gemini generation config exposes penalties and stop sequences but no per-token bias map. Providers that expose an OpenAI-compatible endpoint may accept the field in the request and ignore it, which is worse than rejecting it — your requests succeed, your bias does nothing, and the only symptom is a slow drift in output distribution that no test catches.

So the first question of the migration is not “how do I express this on the new provider”. It is “what was this bias map for”, because the answer usually points at a primitive that both providers have.

The four real uses and what replaces each

Constraining the answer to a fixed set. Biasing the tokens for “yes” and “no” upward, or biasing everything else down, to force a classification label. This is the most common use and it has a strictly better replacement: a JSON Schema with an enum, enforced by the provider’s structured-output mode. That constrains the whole output rather than nudging individual logits, it fails loudly rather than quietly, and it works across providers that support schema-constrained output. The library covers the mechanism in JSON mode versus structured outputs.

Preventing a specific completion from continuing. Biasing a token to -100 so generation avoids a path. If what you actually want is generation to halt at a string, that is the stop-sequence parameter, which exists on every major API — OpenAI stop, Anthropic stop_sequences, Gemini stopSequences — with different limits on how many you may supply. A stop sequence matches text, not tokens, so it does not have the tokenisation problem at all.

Discouraging repetition. A hand-built bias map that penalises tokens already seen is a worse version of the frequency and presence penalties, which are parameters in their own right and are widely available. If your map was generated per-request from the prompt, this is almost certainly what it was doing; delete it and set the penalty.

Suppressing unwanted vocabulary. Brand names, competitor mentions, a phrase legal asked you to remove. This is the use with no clean replacement, and it is also the use that was never working properly for the tokenisation reason above. The honest replacement is an output check: generate, scan the text with a regular expression or a small classifier, and regenerate or redact on a hit. That is deterministic, it is testable, and it is provider-independent. It costs a second pass over the output and occasionally a retry.

If you genuinely need token-level control

Some workloads really do need per-token intervention: constrained decoding for a grammar the provider’s schema mode cannot express, or research work that manipulates the distribution directly. Hosted APIs are the wrong layer for that, and no amount of mapping will make them the right one. Serving the weights yourself puts the sampler back in your process, where bias, banned-token lists and grammar-constrained decoding are all available as first-class controls in the common inference servers.

That is a real cost — hardware, an on-call rotation, a model you now have to update yourself — and it is worth taking only when the token-level control is load-bearing for the product rather than a convenience inherited from an earlier design. For everything else, the replacement table above is the migration. Related: what to do when the same migration also drops logprobs support, which usually travels with logit bias in the same codebase.