A Compatibility Test Matrix Across OpenAI-Compatible Providers

“Does that provider support forced tool choice?” is answered, in most teams, by whoever tried it last and remembers. A matrix turns it into a file: one row per provider, one column per feature, regenerated by a suite and reviewed as a diff when it changes.

The folklore problem

Every codebase that talks to more than one inference provider grows a set of conditionals nobody can justify. A if (provider === "x") that skips structured outputs, added during an incident eight months ago, is still there after the provider shipped support. Somewhere else a feature is used unconditionally because it worked on the one provider it was written against, and it silently degrades on the other two.

Both are the same failure: capability knowledge living in comments and memory rather than in something that gets checked. The fix is not documentation, because documentation goes stale in the same direction as the conditionals. It is a generated file that a test run rewrites and a human reviews.

There is a second reason to write this down, which is that “supported” is not one question. A provider can accept a JSON schema and enforce it; accept it and treat it as a suggestion; accept it and enforce only the top-level type; or reject unfamiliar keywords inside it. Those are four different behaviours that a single boolean in somebody’s head cannot hold, and the difference decides whether your parser needs a repair path. Writing the probe forces you to say which of the four you are asking about, and that alone removes most of the ambiguity the folklore was carrying.

The shape of the matrix

Keep it as JSON rather than a rendered table, so a diff is readable and the file can be imported by code. Three states per cell for the reason given on the single-provider suite — supported, ignored, rejected — plus a fourth, unknown, for a probe that could not run because the provider was unreachable. Collapsing unknown into ignored is how a matrix ends up asserting a provider lost a feature during an outage.

{
  "generatedAt": "2026-08-11",
  "providers": {
    "openai": {
      "model": "gpt-4.1-mini",
      "json_schema": "supported",
      "tool_choice_required": "supported",
      "parallel_tool_calls": "supported",
      "seed": "supported",
      "logprobs": "supported",
      "stream_usage": "supported",
      "n_gt_1": "supported"
    },
    "ollama": {
      "model": "llama3.2",
      "json_schema": "supported",
      "tool_choice_required": "ignored",
      "parallel_tool_calls": "unknown",
      "seed": "supported",
      "logprobs": "ignored",
      "stream_usage": "supported",
      "n_gt_1": "ignored"
    }
  }
}

Record the model alongside the provider. Capability is a property of the pair, not of the vendor — the same account can have one model that honours a strict JSON schema and one that does not, and a matrix keyed only on provider name will average the two into a lie.

Which features are worth a column

A column earns its place if your code would behave differently based on the answer. That is a much shorter list than the API reference.

• Strict schema output. Whether the provider enforces a JSON schema or merely produces JSON-ish text. The difference decides whether you need a repair path; the two are not the same feature.
• Forced tool choice. Whether tool_choice can compel a specific function. If it cannot, any flow that assumes a tool call on the first turn needs a retry.
• Parallel tool calls. Whether one assistant message can carry several tool_calls entries, which decides whether your executor needs to handle a list.
• Usage on a stream. Whether a final chunk carries usage. Without it your cost accounting on streamed requests has to be estimated rather than read.
• Seeded sampling. Whether seed narrows variance at all. Useful to know before you build a test that depends on it.

Resist adding a column for a feature you do not use. Every column is a probe that costs a request against every provider on every run, and a matrix that takes four minutes gets moved to a nightly job and then ignored.

One column that looks tempting and is not worth having is context window. It is real, it varies between providers serving the same model name, and it is exactly the kind of thing a matrix ought to record — but probing it means sending a prompt near the limit, which is expensive on every run and produces a number the provider can change without changing the model. Read it from the provider’s model listing endpoint instead and keep it in a different file, so the capability matrix stays a file about behaviour rather than about quotas.

Generating and committing it

Vitest can compare an object against a file on disk and update that file when you pass the update flag, which is exactly the ergonomics you want: the test fails on a change, and regenerating is one command.

// matrix.test.ts
import { expect, it } from "vitest";
import { probeProvider } from "./probe-provider";

const PROVIDERS = [
  { name: "openai", baseURL: "https://api.openai.com/v1", model: "gpt-4.1-mini" },
  { name: "ollama", baseURL: "http://localhost:11434/v1", model: "llama3.2" },
];

it("matches the committed capability matrix", async () => {
  const providers: Record<string, unknown> = {};
  for (const p of PROVIDERS) providers[p.name] = await probeProvider(p);

  await expect(JSON.stringify({ providers }, null, 2) + "\n")
    .toMatchFileSnapshot("./capability-matrix.json");
}, 120_000);

Leave generatedAt out of the compared object even though the example above shows it in the artefact — a timestamp in a snapshot makes every run a diff. If you want the date recorded, write it in a separate file the test does not compare. And give the test an explicit timeout: the default is measured in seconds and this test makes dozens of live requests.

Gating your code on a cell

The matrix is worth more if the application reads it rather than duplicating its knowledge. Import the JSON, and write the fallback path as a lookup instead of a provider name comparison.

import matrix from "./capability-matrix.json";

export function wantsStrictSchema(provider: string): boolean {
  return matrix.providers[provider]?.json_schema === "supported";
}

Now the day a provider ships strict schemas, one regenerated file removes a repair path from production and the diff says so in one line. That is the whole point: the capability lives in one place, it is proved by a request rather than asserted by a comment, and a change to it is reviewable. Pair it with what happens when a model changes underneath you, which is the same class of drift arriving through a different door.