A Staging Configuration That Mirrors Production Model Settings

Staging exists to be wrong in the same ways production is wrong. Every inference setting that differs between them is a class of bug staging cannot find, and the settings that differ are almost never the ones anybody chose deliberately.

What must be identical

The list is short and each entry is on it because a difference there produces a failure staging cannot reproduce.

• The fully qualified model id. Not the family, not an alias — the exact string, including any snapshot or version suffix. An alias resolves to different weights in different accounts and at different times, which makes it the single most common source of this problem.
• Sampling parameters. Temperature, top-p, top-k, frequency and presence penalties, and the seed policy. A staging environment running at temperature 0 while production runs at 0.7 is testing a system nobody uses; the reverse hides variance until it is in front of customers.
• The system prompt and its template inputs. Not just the template — the rendered result, for the same inputs. A staging tenant with a shorter prompt changes the token budget, the position of everything after it, and the model’s behaviour.
• Tool definitions. The names, descriptions and schemas the model sees. A tool present in one environment and absent in the other changes which tool the model picks in ways that are not local to that tool.
• The token caps and timeouts. A staging environment with a generous max_tokens will never show you the truncation production hits.
• The routing and fallback policy. If production falls back to a second model under load and staging does not, then the fallback path has never been executed anywhere you could watch it.

What must differ

A short allowlist, and everything on it is about isolation rather than behaviour: API keys and credentials, the database and vector index the environment points at, the spend cap, the log destination, the webhook targets, and any feature flag deliberately under test. Rate limits are a borderline case — a lower ceiling in staging is reasonable, but if it is low enough that staging never exercises your concurrency path, you have moved a behaviour into the allowlist by accident.

The point of writing the allowlist down is that it is finite. Anything not on it that differs is drift, and drift is the thing the test in a moment is looking for. Without the list, every difference has a plausible-sounding local justification and the environments diverge one reasonable decision at a time.

One artefact, two overlays

The structural fix is to stop expressing the two environments as two files. Keep one inference configuration — model, sampling, prompts, tools, caps — and let each environment supply only the allowlisted overrides. Then a setting cannot differ unless somebody adds it to the overlay, which is a reviewable act rather than an accident of two files edited at different times.

// config/inference.ts
export const inference = {
  model: "gpt-4.1-mini-2025-04-14",   // pinned snapshot, not an alias
  temperature: 0.2,
  topP: 1,
  maxCompletionTokens: 1200,
  requestTimeoutMs: 30_000,
  promptVersion: "checkout-assistant@7",
  tools: ["lookup_order", "start_return"],
  fallback: { model: "claude-haiku-4-5", after: "timeout" },
} as const;

// Only these may be overridden per environment.
export const ENV_OVERRIDABLE = [
  "apiKey", "baseUrl", "databaseUrl", "vectorIndex",
  "spendCapUsd", "logSink", "webhookUrl",
] as const;

The test that proves it

The assertion is on the resolved configuration — what the process would actually send — rather than on the files. Resolve both environments in one test process and diff them, allowing only the allowlisted keys to differ.

import { expect, it } from "vitest";
import { resolveConfig } from "../src/config";
import { ENV_OVERRIDABLE } from "../config/inference";

function diffKeys(a: Record<string, unknown>, b: Record<string, unknown>) {
  return Object.keys({ ...a, ...b }).filter(
    (k) => JSON.stringify(a[k]) !== JSON.stringify(b[k]),
  );
}

it("staging and production differ only where they are allowed to", () => {
  const staging = resolveConfig("staging");
  const production = resolveConfig("production");

  const unexpected = diffKeys(staging, production)
    .filter((k) => !ENV_OVERRIDABLE.includes(k as never));

  expect(unexpected, "unexpected drift: " + unexpected.join(", ")).toEqual([]);
});

Run it on every commit. It costs nothing, it needs no network, and it fails at review time rather than at 2am — which is the entire value proposition, because the alternative is discovering the difference from a production incident and then arguing about which environment was right.

Where drift comes back

Three routes, all of them around the config file rather than through it. A model alias that resolves differently per account, which is invisible in a diff because both environments say the same string — the fix is pinning the snapshot and asserting the id that comes back on the response. A dashboard setting changed by hand during an incident and never reflected in code. And a prompt stored in a database, where staging and production have different rows and nobody considers that configuration at all.

All three are caught by the same second-order habit: record the resolved settings with each response — model id, prompt version, the sampling parameters actually sent — and compare the recordings between environments rather than the intentions. That also gives you the evidence you need when staging passes and production does not, which is the same problem observed after the fact.

There is one legitimate objection to all of this, and it is worth answering rather than dismissing. Making staging identical to production means paying production model prices for test traffic, which for a frontier model and a busy staging environment is not trivial. The answer is to reduce the volume rather than the fidelity: fewer requests against the real configuration is a smaller staging environment, whereas cheaper requests against a different configuration is a different system. Where the budget genuinely will not stretch, make the cheaper model an explicit, named, allowlisted difference — so that when something behaves differently in production, the first thing anybody reads is a line saying the models are not the same.

The same reasoning applies to data. A staging environment pointed at a synthetic corpus is fine; one pointed at a corpus an order of magnitude smaller than production is not, because retrieval quality, prompt length and truncation behaviour all move with corpus size. If you cannot copy the data, at least match its distribution — document length, language mix, the proportion of records with missing fields — because those are the inputs the prompt actually sees.