A GitHub Actions Matrix for Testing Against Multiple Providers

The matrix is the easy half. The hard half is deciding what a single test can assert when the same prompt goes to four models that were never going to produce the same sentence.

The matrix

A matrix job runs the same steps once per combination of the variables you list, in parallel. For provider testing you want one dimension, not a cross-product, and each entry needs several correlated values — the provider name, the model ID, the name of the secret holding the key. That is what include is for: it adds fully specified entries rather than multiplying dimensions.

name: provider-contract
on:
  pull_request:
    paths: ["src/llm/**", "eval/contract/**", ".github/workflows/provider-contract.yml"]
  schedule:
    - cron: "17 5 * * *"

jobs:
  contract:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    strategy:
      fail-fast: false
      max-parallel: 3
      matrix:
        include:
          - provider: openai
            model: gpt-5.5
            secret: OPENAI_API_KEY
          - provider: anthropic
            model: claude-sonnet-4-5
            secret: ANTHROPIC_API_KEY
          - provider: google
            model: gemini-2.5-pro
            secret: GEMINI_API_KEY
    name: contract (${{ matrix.provider }})
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with: { node-version: 22, cache: npm }
      - run: npm ci
      - run: npx vitest run eval/contract --reporter=github-actions
        env:
          PROVIDER: ${{ matrix.provider }}
          MODEL: ${{ matrix.model }}
          PROVIDER_API_KEY: ${{ secrets[matrix.secret] }}

Two settings do real work here. fail-fast: false overrides the default behaviour of cancelling every in-progress matrix job as soon as one fails — without it, one provider having a bad afternoon destroys the results for the others and you learn nothing. max-parallel caps how many run at once, which matters less for rate limits (the limits are per-provider and these jobs each hit a different one) than for keeping a small runner allocation from being entirely consumed by one workflow.

The name: expression is worth setting. The default job name for an include-based matrix is a rendering of the whole entry, which makes the required-status-check list in branch protection unreadable and, worse, changes whenever you add a field to the entry — silently orphaning the check name you had marked as required.

Getting the right key into the right job

The compact form above uses index access on the secrets context, so the matrix carries the name of the secret and the job reads it. This keeps each job holding exactly one credential, which is the property you want: a job that has all four keys in its environment leaks all four if any step misbehaves.

If your setup will not allow that indirection, the explicit alternative is to map every key into the environment and have the adapter select by provider name. It is simpler to read and strictly worse for blast radius, so prefer it only where you have to:

        env:
          PROVIDER: ${{ matrix.provider }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}

Either way, this workflow cannot run on a pull request from a fork, because forked pull requests receive no secrets at all. That is not a configuration mistake to work around casually — see secrets management for a prompt test suite for what to run instead in that case.

What can be asserted across providers

This is the part that decides whether the workflow is useful. The suite must assert only on things that are genuinely part of your contract with every provider, and the text of the answer is not one of them. Asserting an exact sentence produces a suite that fails on every provider except the one it was written against, and the usual outcome is that somebody adds provider-specific expected strings, at which point it has stopped being a contract test.

What survives the crossing:

• Structural validity. When you ask for JSON against a schema, does the parsed output validate? This is close to binary and it is the single highest-value cross-provider assertion, because structured-output support is exactly where providers differ most. See structured output support.
• Which tool was selected. Given a prompt that unambiguously calls for lookup_order, every provider should select lookup_order. The arguments will vary in formatting; the name should not. Assert the name and validate the arguments against the schema rather than comparing them literally.
• The normalised finish reason. Your adapter maps each provider’s termination signal onto your own vocabulary. Assert on your vocabulary — a truncated response must surface as truncation everywhere, which is the thing most likely to be silently mishandled for a newly added provider.
• Error mapping. Deliberately send an over-long request, a bad key, and a request for a model that does not exist. Each must raise your own error class. This catches the common bug where one provider’s auth failure is classified as retryable and retried three times before failing.
• Refusal and redaction invariants. If your system must never echo a credit card number back, that is a property of the system, not of the model, and it should hold on all of them.
• Streaming shape. Consume a stream and assert that the reassembled result equals the non-streamed result for the same input and seed, and that your adapter emitted the same event sequence regardless of provider.

What legitimately differs

Some things will differ and should not be asserted at all, or should be asserted per-entry with a value carried in the matrix.

• Token counts. Different tokenisers give different numbers for identical text. A shared assertion on token usage will fail for a reason that is not a defect.
• Latency. Put a per-entry budget in the include block if you want to assert it at all, and set it loosely — a CI runner’s network is not your production network.
• Quality scores. A judged-quality threshold that is right for your primary model is arbitrary for a fallback. Carry the threshold in the matrix entry, or keep quality out of this workflow entirely and leave it to the main eval suite.
• Capability presence. Not every model supports every feature. Rather than skipping tests by provider name in the test body, put a capability list in the matrix entry and let the suite skip on capability. Adding a provider then means adding one row, not editing a dozen conditionals.

One provider down is not a red build

If a provider is having an incident, a red required check on every open pull request is noise that trains people to merge past the gate. Use continue-on-error: true on entries you have designated as non-blocking — a provider you support as a fallback but do not route to by default — and gate the merge on a summary job that inspects the results and requires only the entries you consider load-bearing.

The distinction to preserve is between “this provider returned an error” and “this provider returned something that broke our contract”. The first is an outage and should be visible without blocking; the second is a bug in your adapter and should block. Your suite can tell them apart because it already classifies errors — which is one of the assertions above, and the reason this workflow is worth more than a status page subscription.