A Compatibility Test Suite for a Self-Hosted vLLM Endpoint

“OpenAI-compatible” means the endpoint paths and the common fields match closely enough that the official SDK works. It does not mean every parameter is honoured, every response field is populated, or that the surface will be the same after the next upgrade. A compatibility suite is how you find out which of those assumptions your code depends on.

What OpenAI-compatible does and does not mean

vLLM documents a substantial OpenAI-compatible surface — /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models, /v1/responses, /v1/audio/transcriptions among others — alongside endpoints with no OpenAI equivalent such as /tokenize, /detokenize, /version, /health and /metrics. It also documents specific divergences: the suffix parameter on the completions API is not supported, and the user parameter on the chat API is ignored.

The divergences that hurt are not usually the documented ones. They are the parameters your code sends that the server silently accepts and does not act on, and the fields your code reads that the server does not always populate. A parameter that is ignored rather than rejected produces no error anywhere — it produces different output, months later, when somebody wonders why the seed does not seem to do anything.

The other reason to have this suite is upgrades. vLLM’s structured-output request fields are a concrete example: the guided_json, guided_regex, guided_choice and guided_grammar extra_body fields are documented as removed in v0.12.0 in favour of a single structured_outputs object. Code written against the old names stops working on upgrade, and a suite that exercises the field is the difference between finding that in CI and finding it in production.

Discovery: pin the server you are testing

Start every run by recording what you are talking to. A compatibility result is meaningless without the version it was produced against.

# conftest.py
import os, httpx, pytest
from openai import OpenAI

BASE = os.environ["VLLM_BASE_URL"]        # e.g. http://localhost:8000/v1

@pytest.fixture(scope="session")
def server_info():
    root = BASE.rsplit("/v1", 1)[0]
    health = httpx.get(root + "/health", timeout=5)
    assert health.status_code == 200, f"server not healthy: {health.status_code}"
    version = httpx.get(root + "/version", timeout=5).json()
    print("vLLM version under test:", version)
    return version

@pytest.fixture(scope="session")
def client():
    return OpenAI(base_url=BASE, api_key=os.environ.get("VLLM_API_KEY", "EMPTY"),
                  max_retries=0, timeout=60.0)

@pytest.fixture(scope="session")
def model(client):
    models = client.models.list().data
    assert models, "/v1/models returned nothing"
    return models[0].id

Reading the model id from /v1/models rather than hard-coding it is not a nicety. vLLM serves under the id it was started with, which for a local checkpoint is often a filesystem path, and a suite that hard-codes a friendly name fails everywhere except the machine it was written on. Setting max_retries=0 matters too: with the SDK’s documented default of two retries, a test asserting that a bad request fails will still fail, but a test asserting how many requests were made will not mean what you think.

The generation contract

Assert on structure, never on the text. These are the fields your code reads, so these are the fields to pin.

def test_chat_completion_shape(client, model):
    r = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": "Reply with the single word: ok"}],
        max_tokens=8,
        temperature=0,
    )
    assert r.choices, "no choices returned"
    choice = r.choices[0]
    assert choice.message.role == "assistant"
    assert isinstance(choice.message.content, str)
    assert choice.finish_reason in {"stop", "length", "tool_calls", "content_filter"}
    assert r.usage is not None, "usage not populated"
    assert r.usage.prompt_tokens > 0
    assert r.usage.total_tokens == r.usage.prompt_tokens + r.usage.completion_tokens

def test_max_tokens_is_enforced(client, model):
    r = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": "Count slowly from one to fifty."}],
        max_tokens=5,
    )
    assert r.choices[0].finish_reason == "length"
    assert r.usage.completion_tokens <= 5

def test_streaming_yields_usage_when_asked(client, model):
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": "Say hello."}],
        max_tokens=16,
        stream=True,
        stream_options={"include_usage": True},
    )
    chunks = list(stream)
    assert len(chunks) >= 2, "stream produced a single chunk"
    assert any(c.choices and c.choices[0].delta.content for c in chunks)
    finals = [c for c in chunks if c.usage is not None]
    assert finals, "no usage chunk: stream_options.include_usage not honoured"

The last test is the one that earns its keep. Cost accounting from streamed responses depends entirely on a final chunk carrying usage, and if the server does not emit it your spend tracking silently reports zero for every streamed request. That is a failure with no error attached to it, which makes it exactly the kind a compatibility suite exists to surface. Note that this asserts a property of the server you have deployed, not a claim about what any version supports — the test is the claim.

Add a determinism test only if you rely on determinism. Send the same request twice with temperature=0 and a fixed seed, and assert the outputs match. If they do not, that is worth knowing explicitly — batching and floating-point non-associativity on GPU can make identical requests diverge, and a test that documents this for your deployment is more useful than an assumption. If it does not hold, mark the test xfail with a comment rather than deleting it, so the next upgrade tells you if it changes.

Structured output and tool calling

This is where compatibility is thinnest and where most integrations break on upgrade. Two mechanisms, tested separately.

import json, jsonschema

SCHEMA = {
    "type": "object",
    "properties": {
        "category": {"type": "string", "enum": ["billing", "technical", "other"]},
        "confidence": {"type": "number"},
    },
    "required": ["category", "confidence"],
    "additionalProperties": False,
}

def test_structured_outputs_json_schema(client, model):
    # Current field name. On servers before v0.12.0 this was extra_body
    # {"guided_json": SCHEMA}; if this test fails with a 400 mentioning an
    # unknown field, check which surface your pinned version documents.
    r = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": "My card was declined twice."}],
        max_tokens=64,
        extra_body={"structured_outputs": {"json": SCHEMA}},
    )
    payload = json.loads(r.choices[0].message.content)   # must parse
    jsonschema.validate(payload, SCHEMA)                  # must validate

def test_tool_call_shape(client, model):
    tools = [{
        "type": "function",
        "function": {
            "name": "lookup_order",
            "parameters": {
                "type": "object",
                "properties": {"order_id": {"type": "string"}},
                "required": ["order_id"],
            },
        },
    }]
    r = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": "Where is order A-4192?"}],
        tools=tools,
        tool_choice="auto",
        max_tokens=128,
    )
    calls = r.choices[0].message.tool_calls
    assert calls, ("no tool call: the server may need --enable-auto-tool-choice "
                   "and a --tool-call-parser for this model")
    call = calls[0]
    assert call.function.name == "lookup_order"
    args = json.loads(call.function.arguments)            # must be valid JSON
    assert "order_id" in args

Three things are being asserted and they are all structural. The JSON parses; it validates against the schema you supplied; the tool name is one you declared and its arguments deserialise. Nothing asserts what category the model chose, because that is model behaviour and not a compatibility property — a suite that pinned category == "billing" would fail on a model swap and teach everyone to ignore it.

The tool-calling assertion message is doing real work. vLLM documents that automatic tool choice requires the server to be started with --enable-auto-tool-choice and an appropriate --tool-call-parser for the model in question, so an empty tool_calls array is far more often a server-flag problem than a model problem. Putting that in the failure message saves the next person an hour.

Recording the gaps as tests

The most valuable half of this suite is the part that asserts what does not work. A known gap recorded as a passing test is documentation that cannot go stale; the same gap recorded in a wiki page is wrong within two releases.

• Documented divergences. The ignored user parameter and the unsupported suffix parameter are stated in vLLM’s documentation; assert the behaviour you observe, so an upgrade that starts honouring them tells you.
• Parameters your code sends. Enumerate every field your application actually sets — logprobs, n, stop, presence_penalty, seed, response_format — and write one test per field asserting the observable effect, not merely that the request succeeded. Silent acceptance is the failure mode.
• vLLM-specific extras. If you send anything through extra_body that the OpenAI API has no equivalent for, that call is by definition not portable. Keep those tests in a separate file so the portability boundary is visible in the directory listing.
• Errors. Send a nonexistent model id, an oversized prompt, and a malformed tool schema, and assert the status codes. Your failover and retry logic branches on these, and an endpoint that returns 500 where you expected 400 will be retried when it should not be.

Run the whole suite against a candidate version before upgrading, and against production after. Treat a newly failing test as a release note rather than a defect: it is telling you the surface moved, which is precisely what you built it to detect.