Running the Same Contract Suite Against Every Provider You Support

One suite over N providers sounds like a loop. The part that is not a loop is what happens when a provider cannot do something and is right not to — and getting that wrong turns the suite into noise or into a false pass.

The suite as an admission gate

Once you support more than one provider, the contract suite stops being a migration tool and becomes policy. A provider is in the routing pool if and only if it passes; adding one means making it pass first, and a provider that stops passing is removed from the pool rather than discussed. That framing is worth adopting explicitly, because the alternative is a standing argument about whether a particular divergence is acceptable, held under time pressure, every time.

It also gives you an answer to a question that otherwise has none: what does “we support provider X” mean? Without a suite it means somebody once got it working. With one it means a named list of assertions that hold, checked on a schedule, with a date on the last green run. That is the difference between a claim and a fact, and it is the thing you want when a customer asks whether they can pin their traffic to a particular vendor.

The assertions themselves are the ones from the pre-switch suite. What changes at N providers is not the content but the structure: the parameterisation, the capability declarations, and the reporting.

The capability matrix

Providers differ in what they support, and most of those differences are legitimate rather than broken. An open-weights host may not implement structured outputs. An embedding endpoint may not accept the dimensions parameter, which is documented as available only on newer OpenAI models. A small provider may not implement stream_options, or prompt caching, or parallel tool calls.

There are two bad ways to handle this and they fail in opposite directions. Writing a separate suite per provider means the shared assertions drift apart and the exercise loses its point. Deleting from the shared suite everything that any provider fails leaves you asserting the intersection, which for five providers is almost nothing — and it means the day your main provider breaks structured outputs, no test covers it.

The working answer is a declared capability matrix, checked into the repository next to the suite. Each provider entry lists its base URL, its model names, and an explicit set of capabilities. Tests are tagged with the capability they require, and the runner decides per provider whether the test applies. The matrix is a document as much as a fixture: it is the honest answer to “what can this provider actually do”, in one place, in version control, with a history of when each answer changed.

// contract/providers.ts
export type Capability =
  | "streaming" | "stream_usage" | "tools" | "parallel_tools"
  | "structured_outputs" | "embeddings" | "embedding_dimensions";

export const PROVIDERS = [
  {
    name: "primary",
    baseURL: process.env.PRIMARY_URL!,
    apiKey: process.env.PRIMARY_KEY!,
    chatModel: "gpt-4o-mini",
    embedModel: "text-embedding-3-small",
    capabilities: new Set<Capability>([
      "streaming", "stream_usage", "tools", "parallel_tools",
      "structured_outputs", "embeddings", "embedding_dimensions",
    ]),
  },
  {
    name: "openweights-host",
    baseURL: process.env.OWH_URL!,
    apiKey: process.env.OWH_KEY!,
    chatModel: "llama-3.3-70b-instruct",
    embedModel: "bge-large-en-v1.5",
    capabilities: new Set<Capability>(["streaming", "tools", "embeddings"]),
    // no stream_usage: token counts are absent from streamed responses.
    // Reviewed 2026-08-11; recheck when they ship stream_options.
  },
] as const;

Skip, expected-fail, or fail

Three outcomes, and choosing between them for each capability is the real work.

• Skip when the provider does not claim the capability and you do not route that kind of traffic to it. In Vitest this is test.skipIf; in pytest it is pytest.mark.skipif with a reason. Always give the reason a string, because a skip without one becomes permanent by forgetfulness.
• Expected-fail when the provider claims the capability but is known to be broken, and you are waiting on a fix. This is test.fails in Vitest and pytest.mark.xfail in pytest, and its value is the inversion: it goes red when the provider fixes the bug, which is your prompt to remove the workaround you built. A skip never tells you that. Use this sparingly and put a ticket reference in the reason.
• Fail for everything in the baseline — the envelope, usage arithmetic, finish reasons, error status codes. No provider gets an exemption from these, because a provider that cannot report a truncation as length is not OpenAI-compatible in any sense you can build on.

Report skips visibly. A run that says “120 passed” when 40 of the 160 were skipped for a provider you route production traffic to is a false pass, and the false pass is the failure mode this whole structure exists to prevent. Print the skip count per provider in the summary, and treat a rising skip count as a regression in its own right.

import { describe, it, expect } from "vitest";
import OpenAI from "openai";
import { PROVIDERS } from "./providers";

describe.each(PROVIDERS)("$name", (p) => {
  const client = new OpenAI({ baseURL: p.baseURL, apiKey: p.apiKey });
  const has = (c: string) => p.capabilities.has(c as never);

  // baseline — no provider is exempt
  it("reports truncation as length", async () => {
    const res = await client.chat.completions.create({
      model: p.chatModel,
      messages: [{ role: "user", content: "Count from 1 to 200, one per line." }],
      max_tokens: 8,
    });
    expect(res.choices[0].finish_reason).toBe("length");
  });

  // capability-gated
  it.skipIf(!has("stream_usage"))("returns usage on a streamed request", async () => {
    const stream = await client.chat.completions.create({
      model: p.chatModel,
      messages: [{ role: "user", content: "Say ok." }],
      max_tokens: 16,
      stream: true,
      stream_options: { include_usage: true },
    });
    let usage: unknown;
    for await (const chunk of stream) if (chunk.usage) usage = chunk.usage;
    expect(usage).toBeDefined();
  });
});

Shape of the CI job

1. Run providers as a build matrix rather than as a loop inside one job. One failing provider then produces one red cell instead of one red pipeline, and the report tells you which vendor is broken without anybody opening a log.
2. Use fail-fast: false, or whatever your CI calls it. The default of cancelling siblings on the first failure is exactly wrong here: you want the full picture of which providers are healthy, and an outage at one vendor should not hide the state of the others.
3. Give each provider its own credential, scoped to a dedicated test key with its own spend limit where the vendor supports one. A leaked or runaway test key should not be able to spend production budget.
4. Cap the run. A global timeout and a small max_tokens on every request bounds both the wall clock and the bill, and neither assertion in the suite depends on a long completion.
5. Publish the result as a per-provider status that your routing configuration can read, if you want the gate to be automatic rather than advisory. That is the point at which the suite stops being a report and starts being policy.

What it costs to run

The recurring bill is what determines whether this survives, so derive it rather than guessing. Take a suite of nine live chat requests per provider, each with a 400-token prompt and max_tokens of 64 — so at most 3,600 input tokens and 576 output tokens per provider per run. At an illustrative rate of $0.50 per million input tokens and $1.50 per million output tokens, that is 3,600 ÷ 1,000,000 × $0.50 = $0.0018 of input plus 576 ÷ 1,000,000 × $1.50 = $0.00086 of output, or about $0.0027 per provider per run.

Six providers on an hourly schedule is 6 × 24 × $0.0027 ≈ $0.39 a day, under $12 a month. The assumptions in that arithmetic are all in the sentences above — nine requests, 400 input tokens, 64 output tokens, six providers, hourly, and a per-token price that is illustrative and must be replaced with your providers’ own published figures, which differ by more than an order of magnitude across the range of models you might test. The conclusion that survives substitution is structural: the cost scales with providers × frequency × token cap, and the token cap is the term you control most directly.