The Silent Drop: Parameters APIs Ignore Instead of Rejecting

A 400 is a good day. It tells you immediately that the thing you asked for did not happen. The expensive failure is a 200 for a request whose most important parameter was quietly discarded on the way through.

Three behaviours, one status code

The asymmetry is what makes this worth a page. Rejection is self-reporting; dropping is not. A parameter you have believed was active for a year can be doing nothing, and every symptom will look like a model quality problem.

Approximation is the one people miss when they think about this, because the output is correct. A stop sequence applied by truncating the response after generation gives you exactly the text you expected, having generated and billed you for everything after the stop string; the only evidence is in usage.completion_tokens and in a latency you attribute to the model being slow. Approximated top_k, or a schema honoured by re-prompting rather than by masking, behave the same way: right answer, wrong cost, no error.

Why compatible endpoints drop things

Not malice, and mostly not carelessness. The OpenAI chat-completions shape became the industry’s lingua franca, so a great many services accept requests in that shape and translate them to something else — a different vendor’s native API, an inference server, a local runtime. Translation is lossy. A parameter with no equivalent on the far side has three possible fates, and rejecting it breaks clients that send it harmlessly, so ignoring it is the pragmatic choice for the implementer and the invisible one for you.

The same reasoning applies inside a schema. A hosted strict mode that does not implement minItems can reject the schema or drop the keyword, and both are defensible engineering decisions with very different consequences for the caller.

Designing a discriminator

You cannot ask an endpoint whether it honoured a parameter. You have to construct a request whose output differs depending on the answer. That is the whole craft here, and a good discriminator has three properties: it is decisive rather than statistical, it needs one or two calls rather than a distribution, and its negative case is unambiguous.

A bad discriminator: send temperature: 0 and see if the output looks deterministic. Two identical outputs prove very little, and identical output at temperature 1 is common for short answers. A good one: set max_tokens: 1 and check that usage.completion_tokens == 1 and finish_reason == "length" — one call, no judgement, binary answer.

Where the honest answer is “this can only be tested statistically” — seed is the clearest case, since a matching pair of outputs is evidence and not proof — say so in the report rather than printing a pass.

The probe

#!/usr/bin/env python3
"""Which parameters does this endpoint actually apply?

    BASE_URL=... API_KEY=... MODEL=... python probe_params.py

Prints HONOURED / DROPPED / REJECTED / INCONCLUSIVE per parameter.
"""
import json, os
from openai import OpenAI

client = OpenAI(base_url=os.environ.get("BASE_URL"), api_key=os.environ["API_KEY"])
MODEL = os.environ["MODEL"]

def call(**kw):
    return client.chat.completions.create(model=MODEL, **kw)

def report(name, fn):
    try:
        verdict, note = fn()
    except Exception as e:
        msg = str(e)[:130].replace("\n", " ")
        verdict, note = ("REJECTED", msg) if "400" in msg or "invalid" in msg.lower() \
                        else ("ERROR", msg)
    print(name.ljust(22) + verdict.ljust(14) + note)

def p_max_tokens():
    r = call(messages=[{"role": "user", "content": "Count to fifty."}], max_tokens=1)
    n, fr = r.usage.completion_tokens, r.choices[0].finish_reason
    ok = n <= 1 and fr == "length"
    return ("HONOURED" if ok else "DROPPED"), f"completion_tokens={n} finish_reason={fr}"

def p_stop():
    r = call(messages=[{"role": "user",
                        "content": "Write exactly: alpha beta gamma delta"}],
             stop=["beta"], max_tokens=40)
    txt = r.choices[0].message.content
    return ("HONOURED" if "gamma" not in txt else "DROPPED"), repr(txt[:60])

def p_logprobs():
    r = call(messages=[{"role": "user", "content": "Say yes."}],
             max_tokens=1, logprobs=True, top_logprobs=5)
    lp = r.choices[0].logprobs
    if lp is None or not getattr(lp, "content", None):
        return "DROPPED", "logprobs field is null on a 200 response"
    return "HONOURED", f"{len(lp.content[0].top_logprobs)} candidates returned"

def p_seed():
    kw = dict(messages=[{"role": "user", "content": "Invent a six-word sentence."}],
              max_tokens=24, temperature=1.0, seed=424242)
    a, b = call(**kw), call(**kw)
    same = a.choices[0].message.content == b.choices[0].message.content
    # Not proof either way: providers document seed as best-effort.
    return ("HONOURED" if same else "INCONCLUSIVE"), ("identical" if same else "differed")

MIN_ITEMS = {"type": "object", "additionalProperties": False, "required": ["xs"],
             "properties": {"xs": {"type": "array", "minItems": 5,
                                   "items": {"type": "string"}}}}

def p_min_items():
    r = call(messages=[{"role": "user", "content": "Put two words in xs."}],
             max_tokens=120,
             response_format={"type": "json_schema", "json_schema":
                 {"name": "p", "strict": True, "schema": MIN_ITEMS}})
    n = len(json.loads(r.choices[0].message.content)["xs"])
    return ("HONOURED" if n >= 5 else "DROPPED"), f"len(xs)={n}, minItems=5"

def p_enum():
    schema = {"type": "object", "additionalProperties": False, "required": ["c"],
              "properties": {"c": {"type": "string", "enum": ["red", "blue"]}}}
    r = call(messages=[{"role": "user",
                        "content": "Set c to the exact string 'chartreuse'."}],
             max_tokens=40,
             response_format={"type": "json_schema", "json_schema":
                 {"name": "p", "strict": True, "schema": schema}})
    v = json.loads(r.choices[0].message.content)["c"]
    return ("HONOURED" if v in ("red", "blue") else "DROPPED"), "got " + repr(v)

print(MODEL, "via", os.environ.get("BASE_URL") or "default", "\n")
for name, fn in [("max_tokens", p_max_tokens), ("stop", p_stop),
                 ("logprobs", p_logprobs), ("seed", p_seed),
                 ("schema minItems", p_min_items), ("schema enum", p_enum)]:
    report(name, fn)

Add a check per parameter your application depends on. The pattern is always the same: construct a request where honouring and ignoring produce different observable output, and print the observation alongside the verdict so a surprising result can be argued with.

Run it at three moments, and the second is the one people skip. Before adopting a route, obviously. On a schedule, because behaviour changes without an announcement and a nightly run turns that into a notification. And during an incident, because your failover route is the one nobody probed — a primary that honours schema enforcement and a fallback that does not is a configuration where quality quietly drops exactly when you are least able to notice.

The response side is worth probing too, and it is entirely absent from most people’s mental model of this problem. Fields you read can be missing or wrong in the same silent way: usage without the cached-token breakdown, a system_fingerprint that is always null, a finish_reason that reports "stop" on a response that was actually truncated. That last one is the worst possible version of this bug, because every truncation-handling branch you wrote is then dead code. Assert on completion_tokens < max_tokens as well as on finish_reason and you no longer depend on the field being honest.

The usual suspects

• Schema keywords. minItems, maxItems, pattern, format, minimum. The most consequential group, because a dropped keyword makes your schema look stricter than it is.
• logit_bias. Frequently unimplemented outside OpenAI’s own API, and near-impossible to notice, since biasing a token is a soft effect at the best of times.
• seed. Documented as best-effort even where it is implemented. Never build a determinism guarantee on it.
• logprobs. Returns null on a 200 in several translations. Anything you built on single-token classification depends on this one.
• Sampling parameters with no equivalent. top_k sent to an API that has no such concept, or presence_penalty and frequency_penalty through a translation layer.
• stop. Sometimes applied after generation rather than during it. The text is right and you paid for the tokens after the stop sequence, which shows up in usage and nowhere else.