Migrating an Evaluation Harness to a New Provider

You pointed the harness at a new provider, the suite still runs, and the aggregate score fell several points. Before you conclude the new model is worse, rule out the four ways the harness itself can produce exactly that number — because three of them look identical to a capability regression in the summary output.

The symptom

The failure mode that matters here is not a crash. A harness that throws is easy: the stack trace names the field. The dangerous case is the one where every case runs, every case is scored, and the number is lower — because that number is what a migration decision gets made on, and nothing in it flags that the comparison was invalid.

The claim on the table when someone shows you that number is: the same task, scored the same way, produced a worse result. A provider swap can break the “same task” half and the “scored the same way” half independently, and usually breaks both a little.

What makes two runs comparable

Two eval runs are comparable when the only intended difference is the thing under test. Concretely, for a model swap that means the input the model saw is semantically identical, the generation was allowed to finish under the same constraints, the output was parsed by the same rules, and the scorer is unchanged. A provider migration puts pressure on all four:

• The input the model saw. Not the input your harness constructed — the one that arrived after translation. These differ more often than people expect.
• The constraints on generation. Token limits and stop sequences bound the answer, and they are counted in the target model’s tokenizer, not the source’s.
• The parse. Which field holds the text, and how you recognise an incomplete answer.
• The scorer. Fixed, or you have two variables.

Four harness-level causes

The system prompt was dropped

The commonest one, and the one with the largest effect. Some APIs carry the system instruction as a message with a dedicated role inside the message list; others take it as a separate top-level system parameter outside the list. A translator that maps messages element-by-element and ignores anything that is not a user or assistant turn silently discards it — and a model evaluated without its system prompt is being asked a different question. If your score fell a lot rather than a little, check this first: log the fully serialised outbound request body for one case and read it.

The related trap is role vocabulary. OpenAI introduced a developer message role alongside the older system role, and a harness that emits one against an API expecting the other may have the message rejected or, worse, accepted and treated with different precedence.

Truncation is being scored as a wrong answer

The field that tells you generation stopped early is not spelled the same everywhere. OpenAI’s chat completions carry finish_reason with values including stop, length, tool_calls and content_filter; Anthropic’s messages carry stop_reason with values including end_turn, max_tokens, stop_sequence and tool_use. A harness that checks finish_reason == "length" to mark a case invalid finds no such field on the new provider, marks nothing invalid, and scores every truncated answer as a failure.

Two things then compound. The token budget is denominated in the target model’s tokenizer, so the same max_tokens buys a different amount of text — see tokenizer comparison for why the counts diverge. And some models are simply more verbose before reaching the answer, so a budget that was generous becomes tight. Fix: normalise the stop reason into your own enum at the client boundary, and assert on the truncation rate per run rather than discovering it in the score.

A default changed underneath you

Any generation parameter your harness does not set explicitly is supplied by the provider, and the values differ. Temperature is the obvious one; top-p, penalties and the maximum output length also have per-provider defaults, and some parameters exist on one side only. Sampling nondeterminism is not a bug, but it is variance you did not budget for — if your suite is small, a temperature default of 1 on the new side can move the aggregate by more than the model difference you are trying to measure.

Set every parameter explicitly in an eval harness, including the ones whose defaults you believe you know. Where the run needs to be as repeatable as the API allows, see temperature zero for tests — and note that a seed parameter is a capability one provider may have and another may not, which is its own migration problem covered in what to do when a provider has no seed parameter.

The output shape changed and the extractor did not

Harnesses accumulate small extraction hacks: strip a leading “Sure, here is”, take everything after the last colon, pull the first fenced code block, read choices[0].message.content as a string. The last of these is the sharpest, because content is a plain string in one shape and an array of typed blocks in another — and an extractor that stringifies an array of blocks produces something that never matches the expected answer. Where the model returned a tool call rather than text, the text field may legitimately be empty.

When the judge moved too

If your scoring uses an LLM judge and the judge runs on the provider you just migrated, you changed two things at once and the resulting number is uninterpretable. This is worth being blunt about: a model-graded eval where both the candidate and the grader moved cannot attribute the delta to either.

Pin the judge. It should be a named model on a provider you are not migrating, held fixed across both runs, with its own prompt versioned alongside the suite. If the judge must move, migrate it in a separate run with the candidate model held constant, so you get two one-variable comparisons instead of one two-variable one. And keep a subset of cases scored by an exact or programmatic check — string match, schema validation, unit test — because that subset is the anchor that tells you whether the judge drifted at all. More on the failure classes a judge introduces in eval blind spots.

The bisection that finds it

1. Re-run the old provider on the current harness code. If that score also fell, the harness change is the cause and the provider is innocent. This one step resolves a surprising share of cases and costs one run.
2. Dump the serialised request for one case on both sides and diff them by eye. You are looking for a missing system instruction, a dropped parameter, and a different token limit.
3. Tabulate stop reasons for the whole run, normalised into your own enum. A truncation rate that differs between the two runs explains the score before anything about capability does.
4. Score only the programmatic subset with the judge disabled. If that subset is flat and the judged subset moved, the judge is implicated, not the model.
5. Read twenty failures by hand. Not the aggregate — the actual outputs. Harness artefacts are obvious at this resolution and invisible at every higher one: empty strings, half-sentences, JSON with a trailing brace missing, a refusal where you expected an answer.
6. Only then compare capability, on cases that ran clean on both sides, and report the sample size alongside the number.

If step five turns up refusals rather than mangled answers, you have a different problem with a different fix — that one is refusal behaviour after a model migration.