Where OpenAI-Compatible Endpoints Actually Break

The failure this page is about produces no stack trace and no error string. You send seed, the server returns 200, and the outputs are not reproducible. You send n: 3 and get one choice back. Nothing in the response says the parameter was discarded, because discarding unknown parameters is the correct behaviour for a permissive server and there is no way to distinguish it from honouring them.

The symptom: no error, wrong behaviour

The reason this is structural rather than a bug in any one project: the OpenAI request body is open. Servers are expected to ignore fields they do not recognise, because rejecting them would break every client that sends a newer field than the server knows about. That leniency is load-bearing, and its cost is that “accepted” carries no information about “implemented”.

There is no field in the response that reports which parameters were applied. system_fingerprint gets mistaken for this and is not: on OpenAI it identifies the backend configuration so you can detect that determinism guarantees may no longer hold, and most compatible servers either omit it or return a constant. A constant fingerprint does not mean your requests are deterministic.

You will sometimes get a loud failure instead, and those are worth recognising because they identify the class of problem immediately. The literal string "temperature must be between 0 and 1" from a server that accepts the OpenAI schema means it has a narrower valid range than OpenAI’s, so code that sends 1.2 works on one endpoint and 400s on another. An error naming "Invalid Schema" on a response_format request means the server validates JSON Schema more strictly than OpenAI does, or supports only a subset of it. Both are better than silence.

Sampling parameters that get dropped

These are the request fields most often accepted and ignored, roughly in order of how often it happens.

• seed. Requires the server to thread a seed into the sampler and to run batches in a way that does not perturb results. Many servers accept it and do nothing; some honour it only when batch size is one. The test is trivial — two identical requests with the same seed and a non-zero temperature — and worth running, because the whole point of the parameter is a guarantee you cannot verify by inspection. See the page on the seed parameter for what it does and does not promise even where it works.
• n. Multiple completions per request require batched sampling from one prefill. Servers commonly return a single choice regardless, so choices.length is your check. Code that reads choices[0] will never notice.
• logprobs and top_logprobs. Require exposing per-token distributions through the serving stack. Frequently unimplemented; sometimes present but with a different nesting than OpenAI’s, which is worse than absent because it deserialises to an empty structure. Check for a populated logprobs object on the choice, not merely a present one.
• logit_bias. Requires a tokeniser that agrees with the token ids you are sending. Even where implemented, the ids are the server’s tokeniser’s ids, so a bias map built against OpenAI’s vocabulary biases arbitrary unrelated tokens. This one is actively dangerous rather than merely inert.
• stop. Usually implemented, but the details differ: how many sequences are accepted, whether the matched sequence is stripped from the output or left in, and whether matching is done on detokenised text or on token boundaries. A sequence that spans a token boundary may not match at all.
• presence_penalty and frequency_penalty. Widely accepted, variably implemented, and with no observable signature that distinguishes “applied weakly” from “not applied”. If your quality depends on them, they are a risk you should retest after any endpoint change.
• max_tokens versus max_completion_tokens. OpenAI introduced the second name and deprecated the first for newer models. Compatible servers overwhelmingly implement the older name. A client library updated to send only the newer one against an older server results in an unbounded generation, because the cap was in a field the server ignored.

Tool calling, where it breaks worst

Tool calling is the largest surface and the least uniformly implemented, and it fails in ways that look like the model being bad at tool use rather than the server being incomplete.

The first thing to check is which generation of the API the server implements. The original shape used a functions array and returned a function_call object on the message; the current one uses tools and returns a tool_calls array. Sending the modern shape to a server that only implements the old one produces either a 400 or, worse, a response with no tool call at all, because the tools were ignored and the model was never told they existed. If you see a deprecation warning naming function_call and tool_calls, you are on the boundary between the two.

• Parallel tool calls. Many servers return at most one entry in tool_calls even where the model wanted several. OpenAI exposes parallel_tool_calls to control this; a server that ignores the flag and always returns one call will make an agent loop take several turns to do what should take one, at several times the cost.
• tool_choice enforcement. The string forms "auto", "none" and "required" and the object form naming a specific function are four separate features. "required" is the one most often unimplemented, and it fails as an ordinary text answer where your code expected a call.
• Streamed tool calls. In the OpenAI shape, tool call arguments arrive as partial JSON fragments across chunks, keyed by an index within the tool_calls array in the delta, with the call id and function name sent only on the first fragment. Servers get this wrong more often than any other part of streaming — repeating the id on every fragment, omitting the index, or sending the whole arguments object at once. Accumulator code written against the real shape breaks on all three.

Structured output that is only advisory

response_format has two levels and they are frequently conflated. { type: "json_object" } asks for syntactically valid JSON. { type: "json_schema", ... } with strict: true asks for output constrained to a schema, which requires constrained decoding in the serving stack.

A server that implements the first and accepts the second returns JSON that is valid but does not follow your schema — missing required properties, extra properties, wrong types — and does so intermittently, which reads as model unreliability. The check is to send a schema with a required enum and confirm you never get a value outside it across enough samples to matter; a constrained decoder cannot produce one, and a prompt-based implementation will eventually. The distinction is covered in JSON mode versus structured outputs.

Streaming details that go wrong

Beyond tool calls, four streaming defects are common enough to test for by name: a missing data: [DONE] sentinel, so clients cannot distinguish completion from a dropped connection; finish_reason never being set to anything but null, so truncation is undetectable; object being sent as chat.completion instead of chat.completion.chunk, which fails typed deserialisation; and stream_options.include_usage being accepted without the usage chunk ever arriving.

Add one that is not the server’s fault and is diagnosed the same way: a reverse proxy buffering the response, which turns a correct stream into a single delivery at the end. If chunks arrive correctly from the server directly and all at once through your ingress, the server is fine and the proxy configuration is not.

What to do about it

1. Establish the capability set once, not per call site. Probe the endpoint for the parameters you actually use and record the answers in configuration keyed by endpoint. See testing whether an endpoint is really compatible for the probes.
2. Fail loudly at the boundary. If your code sends a parameter the recorded capability set says is unsupported, raise rather than sending it. A silently ignored parameter becomes a visible configuration error rather than a quality regression three weeks later.
3. Shim what can be shimmed, and only that. n can be emulated with concurrent requests, at the cost of re-running prefill each time. A missing stop implementation can be emulated by truncating the streamed text yourself, which costs you the tokens generated after the match. Determinism from seed cannot be emulated at all, and neither can schema-constrained decoding; for those, validate and retry, and budget for the retries.
4. Re-run the probes on every endpoint or version change. These are capabilities of a deployment, not of a protocol. An upgrade of the serving software changes the answers in both directions.