Skip to content
Multigrid

Testing Whether an Endpoint Is Really OpenAI-Compatible

10 min read · updated August 11, 2026

A compatibility test that asserts “the request returned 200” tells you almost nothing, because a server that ignores every optional parameter also returns 200. The test has to check theeffect of each parameter, which means each probe needs a response property that could only hold if the parameter was applied.

Three outcomes, not two

Design the report around three states per feature, because collapsing them loses the only distinction you care about.

  • Implemented — the request succeeded and the response shows the parameter took effect.
  • Rejected — the server returned an error. This is the good failure. You know where you stand, and your code can branch on it at runtime.
  • Ignored — the request succeeded and the response shows no evidence the parameter was applied. This is the state the whole exercise exists to find, and it is invisible to any test that only checks status codes.

A fourth state is worth recording separately: inconclusive. Some probes cannot give a definite answer from a single call — a model that legitimately chose not to emit a tool call, for instance — and a report that reports a guess as a fact is worse than one that admits the gap. Keep the raw response for every inconclusive probe so a human can look.

The probes worth running

Each of these pairs a request with a property that is only true if the server did the work. Keep max_tokens small everywhere; you are testing plumbing, not generation.

  • Envelope. A minimal non-streaming call. Assert object equals chat.completion, that created is a number, that choices[0].message.content is a string and that choices[0].finish_reason is present.
  • Usage. The same response has a usage object with all three counters, each a positive integer. Absent or zero counts mean cost tracking will not work.
  • Model listing. GET /models returns an object with a data array containing the model you just used.
  • Streaming framing. A streamed call. Assert the content type is text/event-stream, that every chunk has object equal to chat.completion.chunk, that finish_reason is non-null on exactly one chunk, and that the final line is literally data: [DONE].
  • Streaming usage. The same call with stream_options asking for usage. Assert a chunk arrives with an empty choices array and a populated usage.
  • Stop sequences. Ask for a count from one to ten with stop set to the string for five. Assert the output does not contain it. This also reveals whether the matched sequence is stripped.
  • Seed. Two identical requests, same seed, temperature above zero, enough tokens to diverge. Identical outputs suggest it is honoured; differing outputs mean it is not. Note this probe is one-sided — identical outputs can happen by chance on a short generation, so make the generation long enough that coincidence is implausible.
  • Multiple completions. n set to three; assert choices.length is three.
  • Logprobs. Request them and assert the returned structure is populated, not merely present.
  • Tools. A single unambiguous tool and a prompt that requires it, with tool_choice set to require a call. Assert tool_calls exists, has an id and a function name, and that its arguments string parses as JSON.
  • Structured output. A schema with a required property whose value is a two-member enum. Assert the parsed output has the property and its value is in the enum.
  • Error shape. Deliberately send an invalid value. Assert a 4xx, and record whether the body has the OpenAI error.message / error.type / error.param shape described in the error shape mapping page.

The script

  1. Install the official client. The point of using the real SDK rather than raw HTTP is that SDK deserialisation is itself part of the test: an envelope that fails to parse is a compatibility failure and you want it to surface as one.
    pip install openai
  2. Save the script below as probe.py. It takes a base URL, a key and a model name, and prints one line per probe.
    import json, os, sys, httpx
from openai import OpenAI, APIStatusError

BASE = sys.argv[1]
MODEL = sys.argv[2]
KEY = os.environ.get("PROBE_API_KEY", "not-needed")
client = OpenAI(base_url=BASE, api_key=KEY, timeout=60.0)

results = []
def record(name, state, detail=""):
    results.append((name, state, detail))

def probe(name, fn):
    try:
        record(name, *fn())
    except APIStatusError as e:
        record(name, "REJECTED", "HTTP " + str(e.status_code))
    except Exception as e:
        record(name, "ERROR", type(e).__name__ + ": " + str(e)[:80])

def envelope():
    r = client.chat.completions.create(
        model=MODEL, max_tokens=16,
        messages=[{"role": "user", "content": "Say hi."}])
    ok = (r.object == "chat.completion"
          and isinstance(r.created, int)
          and r.choices[0].finish_reason is not None)
    return ("IMPLEMENTED" if ok else "IGNORED", "object=" + str(r.object))

def usage():
    r = client.chat.completions.create(
        model=MODEL, max_tokens=16,
        messages=[{"role": "user", "content": "Say hi."}])
    u = r.usage
    ok = u and u.prompt_tokens > 0 and u.completion_tokens > 0
    return ("IMPLEMENTED" if ok else "IGNORED", str(u))

def streaming():
    # Raw HTTP here on purpose: the SDK hides the sentinel, and the
    # sentinel is one of the things under test.
    body = {"model": MODEL, "max_tokens": 24, "stream": True,
            "messages": [{"role": "user", "content": "Count to five."}]}
    saw_done, kinds, finishes = False, set(), 0
    with httpx.stream("POST", BASE.rstrip("/") + "/chat/completions",
                      json=body, timeout=60.0,
                      headers={"Authorization": "Bearer " + KEY}) as resp:
        ctype = resp.headers.get("content-type", "")
        for line in resp.iter_lines():
            if not line.startswith("data:"):
                continue
            payload = line[5:].strip()
            if payload == "[DONE]":
                saw_done = True
                break
            chunk = json.loads(payload)
            kinds.add(chunk.get("object"))
            ch = (chunk.get("choices") or [{}])[0]
            if ch.get("finish_reason"):
                finishes += 1
    ok = (saw_done and kinds == {"chat.completion.chunk"}
          and finishes == 1 and "text/event-stream" in ctype)
    return ("IMPLEMENTED" if ok else "IGNORED",
            "done=" + str(saw_done) + " objects=" + str(kinds)
            + " finish_reasons=" + str(finishes))

def stop_sequences():
    r = client.chat.completions.create(
        model=MODEL, max_tokens=48, stop=["5"],
        messages=[{"role": "user", "content":
                   "Count from 1 to 10, digits separated by spaces."}])
    text = r.choices[0].message.content or ""
    return ("IMPLEMENTED" if "5" not in text else "IGNORED", repr(text[:40]))

def seed():
    def once():
        r = client.chat.completions.create(
            model=MODEL, max_tokens=64, temperature=1.0, seed=1234,
            messages=[{"role": "user", "content":
                       "Invent a two-sentence story about a lighthouse."}])
        return r.choices[0].message.content
    a, b = once(), once()
    return ("IMPLEMENTED" if a == b else "IGNORED",
            "identical" if a == b else "diverged")

def multiple_completions():
    r = client.chat.completions.create(
        model=MODEL, max_tokens=16, n=3, temperature=1.0,
        messages=[{"role": "user", "content": "Name a colour."}])
    return ("IMPLEMENTED" if len(r.choices) == 3 else "IGNORED",
            "choices=" + str(len(r.choices)))

def tools():
    spec = [{"type": "function", "function": {
        "name": "get_weather",
        "description": "Get the weather for a city.",
        "parameters": {"type": "object",
                       "properties": {"city": {"type": "string"}},
                       "required": ["city"]}}}]
    r = client.chat.completions.create(
        model=MODEL, max_tokens=64, tools=spec, tool_choice="required",
        messages=[{"role": "user", "content": "Weather in Oslo?"}])
    calls = r.choices[0].message.tool_calls or []
    if not calls:
        return ("IGNORED", "tool_choice=required produced no call")
    json.loads(calls[0].function.arguments)
    return ("IMPLEMENTED", calls[0].function.name)

def error_shape():
    try:
        client.chat.completions.create(
            model=MODEL, max_tokens=-5,
            messages=[{"role": "user", "content": "hi"}])
    except APIStatusError as e:
        b = e.response.json() if e.response is not None else {}
        err = b.get("error", {}) if isinstance(b, dict) else {}
        shaped = "message" in err and "type" in err
        return ("IMPLEMENTED" if shaped else "IGNORED", json.dumps(b)[:90])
    return ("IGNORED", "invalid max_tokens was accepted")

for name, fn in [("envelope", envelope), ("usage", usage),
                 ("streaming", streaming), ("stop", stop_sequences),
                 ("seed", seed), ("n", multiple_completions),
                 ("tools", tools), ("error_shape", error_shape)]:
    probe(name, fn)

width = max(len(n) for n, _, _ in results)
for name, state, detail in results:
    print(name.ljust(width), state.ljust(12), detail)
  3. Run it against the endpoint under test. The model name has to be one that endpoint actually serves.
    PROBE_API_KEY=sk-... python probe.py https://api.example.com/v1 my-model
  4. Run it against a reference implementation you trust as well, using a model you know supports everything. Probes that report IGNORED on both are probes with a bug in them, not findings. This calibration step is the one people skip and it is what stops the report being fiction.

Reading the report

Read REJECTED as good news: a server that errors on n: 3 has told you the truth, and your client can handle it. Read IGNORED as a finding that needs a decision — shim it, avoid the parameter, or accept the loss and record why.

Three probes need care in interpretation. The seed probe is one-sided: divergent outputs prove the seed is not honoured, identical outputs are only evidence, so make the generation long and repeat it before concluding. The stop probe can fail because the sequence spans a token boundary rather than because stop is unimplemented; try a second sequence before believing it. And the tools probe depends on the model as well as the server — a very small model may fail to produce a call even where the server supports it, which is why tool_choice: "required" is used rather than relying on the model to choose.

Keep the raw response for every probe. The report is a summary and the summary is the thing you will disagree with in three months.

Keeping it honest in CI

The report’s value is not the first run, it is the diff. These are capabilities of a deployment, and a serving-software upgrade changes them in both directions without any announcement you will see.

  1. Commit the report for each endpoint as a checked-in file, one line per probe, sorted.
  2. Run the probes on a schedule — nightly is enough — and fail the job on any diff against the committed file, in either direction. A capability appearing is as much a reason to look as one disappearing, because it usually means the deployment moved.
  3. Assert in your application code against the same file, so a request that uses a parameter recorded as IGNORED raises at the boundary rather than degrading quietly.
  4. Re-run against a reference endpoint in the same job. If both change in the same way, your probe broke; if only one changed, the endpoint did.

The event this catches is the one that has no other detector: the endpoint that quietly stopped honouring a parameter it used to honour, which otherwise reaches you as a gradual quality complaint with no deploy of yours to correlate against. What that costs to diagnose without a probe report is the entire argument for keeping one.

Related