Using the seed Parameter in Tests Against an OpenAI-Compatible API

The seed parameter is the closest thing a hosted model has to a reproducibility control, and every vendor that ships one describes it as best effort. Reading that wording carefully is the difference between a suite that uses seeds well and one that builds an assertion on a promise nobody made.

What a seed is promised to do

A seed fixes the pseudo-random number generator that the sampler draws from. If everything else about the computation is identical, the same seed selects the same tokens. That conditional is doing all the work, because “everything else” includes how your request was batched with other tenants’ requests, which GPU it landed on, and which build of the serving stack was running.

OpenAI’s API reference states that if a seed is specified, the system “will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result”, and that “determinism isn’t guaranteed”. It directs you to system_fingerprint to monitor backend changes. Read the whole sentence in OpenAI’s advanced usage guide.

Google’s Gemini API documents a seed field inside generationConfig with the same shape of promise: with a fixed seed the model makes a best effort to return the same response, and deterministic output is not guaranteed. See Google’s generate-content reference.

It is worth being precise about what that leaves. A seed cannot make two different models agree, cannot survive a change to any other parameter, and cannot compensate for the request being scheduled alongside different neighbours on the same GPU. It is a control over one component of a pipeline with several non-deterministic components, and the vendors word it that way because that is what it is. A suite that treats a seed as a guarantee has taken a probabilistic improvement and built an assertion that requires certainty.

Who documents one, and under what name

• OpenAI Chat Completions — seed, an integer, paired with system_fingerprint on the response. The reproducible-outputs guidance is written against Chat Completions specifically; do not assume the field carries over unchanged to a newer endpoint without checking that endpoint’s own reference.
• Google Gemini — seed inside generationConfig (GenerateContentConfig in the SDKs), best effort.
• Mistral — random_seed, not seed. A test harness that sets seed against Mistral is setting a field the endpoint has no reason to recognise. See Mistral’s API reference.
• Anthropic — no seed parameter on the Messages API. There is nothing to set, and a library that appears to accept one for Claude is dropping it or erroring on your behalf. What to do when a provider does not support seed is the page for this case.
• vLLM and self-hosted OpenAI-compatible servers — seed per request. vLLM’s reproducibility documentation is unusually honest about the limits: the guarantee holds only on the same hardware and the same vLLM version, and batching can still change results unless batch invariance is explicitly enabled.

Proving your endpoint honours it

An OpenAI-compatible endpoint that does not implement seeds has two options: reject the request, or ignore the field and return 200. The second is far more common and far more dangerous, because your suite looks configured and is not. The probe is two identical requests.

# tests/test_seed_support.py
import pytest

PROMPT = [{"role": "user", "content": "Name three prime numbers above 100."}]

def test_endpoint_honours_seed(complete):
    a = complete(PROMPT, seed=1234)
    b = complete(PROMPT, seed=1234)
    c = complete(PROMPT, seed=9999)

    same = a.choices[0].message.content == b.choices[0].message.content
    different = a.choices[0].message.content != c.choices[0].message.content

    # Not a pass/fail gate: a report. Both can legitimately be False.
    print("same seed reproduced:", same, "different seed diverged:", different)
    assert a.model == b.model == c.model

Interpret it like this. Same seed reproducing and different seeds diverging together is the signal that the field is wired through. Same seed reproducing while different seeds also reproduce means you are at temperature 0 and the seed is irrelevant — run the probe at a non-zero temperature to separate the two. Same seed not reproducing means the field is ignored, or the backend changed between the two calls, and system_fingerprint tells you which.

Putting the seed in the harness

Derive the seed from the test identity rather than using one global constant. A constant seed across every case means every case explores the same corner of the sampler; a per-case seed gives you variety across the suite and reproducibility within a case, which is what you actually want.

# tests/conftest.py — deterministic per-test seed
import hashlib
import pytest

@pytest.fixture
def seed(request):
    # request.node.nodeid is stable for a given test, across machines and runs.
    digest = hashlib.sha256(request.node.nodeid.encode()).digest()
    return int.from_bytes(digest[:4], "big")   # a 32-bit integer

Log the seed on failure. A test that fails and does not print the seed it used has thrown away the one thing that might reproduce the failure. Store the seed alongside any recorded expectation for the same reason.

Do not derive the seed from anything that varies between machines or runs — a hash of the file path breaks when the repository is checked out somewhere else, and the wall clock breaks always. The test node id is stable because it is derived from the module path within the project and the test name, which is exactly the identity you want. Where a case comes from a dataset rather than a test function, hash the case id instead, for the same reason and with the same property.

One more thing the harness should do: keep the seed out of the prompt. It is tempting to include a run identifier in the system message so traces are easy to correlate, and it is the same mistake that destroys prompt caching — a prompt that varies per run is not the same request, so the seed has nothing constant to reproduce against. Correlate through metadata fields and headers, never through the message content.

What you may assert once it works

A working seed buys you a reproducible failure, which is worth a great deal during debugging. It does not buy you a licence to assert on exact output text in CI, because the seed does not survive a backend change and asserting equality means a vendor infrastructure update turns into a red build on an unrelated pull request.

The assertions that hold are structural: the response parses, the schema validates, the tool that fires is the one you expected with the arity you expected, the cited identifiers all exist in the input, no redacted field appears in the output. Combining low temperature with property assertions works through that catalogue.