What to Do When a Provider Doesn't Support the seed Parameter

Two different things send people here. One is a 400 naming seed as an unrecognised argument. The other is worse: a 200, no error anywhere, and a test that still returns a different answer every run. The fix is the same shape in both cases, and it is not “find a provider with seeds”.

The two symptoms

The loud one looks like Unrecognized request argument supplied: seed from an OpenAI-compatible endpoint that does not implement the field, or a client-side TypeError: unexpected keyword argument 'seed' from an SDK whose method signature has no such parameter — the Anthropic SDK, for instance, has no seed to pass because the Messages API has none.

The quiet one has no error string at all. The request succeeds, the field is discarded by a proxy or by a permissive server that ignores unknown keys, and the only evidence is that two runs of the same test disagree. This is the case worth building a check for, because nothing else will tell you.

Telling ignored from rejected

Run three requests at a non-zero temperature: two with the same seed, one with a different seed. Temperature must be non-zero for this to mean anything, because at 0 all three converge and the test cannot distinguish a working seed from an ignored one.

# Diagnostic, not a gate. Run it once against each provider you support.
import itertools

def seed_status(call, prompt):
    """Returns one of: honoured, ignored, rejected."""
    try:
        a = call(prompt, seed=7, temperature=0.8)
        b = call(prompt, seed=7, temperature=0.8)
        c = call(prompt, seed=8, temperature=0.8)
    except TypeError:
        return "rejected (sdk has no such parameter)"
    except Exception as exc:
        if "seed" in str(exc):
            return f"rejected ({exc})"
        raise

    if a == b and a != c:
        return "honoured"
    if a == b and a == c:
        return "inconclusive: raise temperature or lengthen the prompt"
    return "ignored"

Record the answer per provider in your repository, next to the model configuration, and let the suite branch on it. Discovering it once and writing it down is much cheaper than discovering it repeatedly through flaky tests.

The inconclusive result deserves attention rather than a shrug. Short prompts with an obvious answer converge regardless of temperature, because the distribution is sharply peaked and every sample lands on the same tokens. If the probe returns inconclusive, lengthen the prompt and ask for something open-ended — a paragraph of prose rather than three numbers — before concluding anything. A probe that cannot distinguish its own outcomes is worse than no probe, because it will be read as a pass.

It is also worth re-running the probe when you change providers, change a gateway, or change a base URL. Seed handling is a property of the endpoint, not of the SDK, and an OpenAI-compatible URL swapped in for a self-hosted server is a different endpoint with different behaviour behind an identical client.

Fallback 1: assert properties, not text

This is first because it is free and because it is usually what you should have been doing anyway. A seed makes an exact-text assertionpossible; it does not make it a good idea, since the assertion still breaks the next time the provider changes its backend. Asserting a property instead makes the seed unnecessary.

• Structure. The response parses as JSON, validates against the schema, every required field is present, every enum value is a member of its enum.
• Grounding. Every identifier, quotation or figure in the output appears in the input. This is a set-membership check and it catches fabrication without asserting a sentence.
• Tool behaviour. The tool called is the expected one, exactly one call was made, the arguments validate against the tool’s own schema.
• Absence. No API key, no system prompt fragment, no redacted field, no forbidden phrase appears in the output.
• Bounds. Output length within a range, at most one recommendation, a numeric score within its scale.

The reason this rung comes first is not only cost. A property assertion is also a better description of the requirement, so it documents the contract to whoever reads the test next. An exact-match assertion says only that somebody once observed this string; it contains no information about which part of it mattered.

Fallback 2: metamorphic relations

A metamorphic relation asserts that two outputs relate to each other in a known way, without saying what either one is. That is exactly the shape of assertion a non-deterministic system can support, and it is the technique most under-used in model testing.

• Reordering the retrieved documents in a RAG prompt should not change which document is cited as the source of a specific fact.
• Renaming an entity consistently throughout the input should produce an output with the new name and the same structure.
• Adding an irrelevant paragraph should not change a classification label.
• Asking for a summary of a longer document should not produce a longer summary than the same request over a strict subset of it.

Each of these is two calls rather than one, so it doubles the cost of the case. That is the trade, and it is usually worth it for a handful of relations rather than for every case.

Fallback 3: repeat and require a quorum

If the assertion genuinely must be about the content, run the case n times and require k agreements. This converts a binary assertion into a statistical one, and it costs n times the tokens, so reserve it for the small set of cases where nothing cheaper works.

import collections

def majority_label(call, prompt, n=5, k=4):
    labels = [classify(call(prompt, temperature=0.7)) for _ in range(n)]
    label, count = collections.Counter(labels).most_common(1)[0]
    assert count >= k, f"no quorum: {collections.Counter(labels)}"
    return label

Two honest caveats. Five calls at temperature 0.7 is five times the bill for that case, and the monthly arithmetic should include it. And a quorum of 4 out of 5 is a threshold you chose, not a measurement; state it in the assertion message so the next person can see it is a policy rather than a fact.

Fallback 4: record the response once

For everything downstream of the model — parsing, validation, business logic, error handling, retry — you do not need the model at all. Record one real response and replay it. The model call becomes deterministic by removal rather than by parameter, which is the strongest form of the fix and also the cheapest. Recording fixtures once covers the pattern and its one real hazard, which is a recording that silently goes stale.

The ladder is deliberately ordered by cost. Properties are free, metamorphic relations double a case, quorum multiplies it by n, and recording is a one-off charge amortised across every future run. A suite that reaches for the quorum first is usually a suite that skipped the first rung.