A Smoke Test That Runs Against the Real API Before Every Deploy

Every test passed. The build shipped. The first user request returns a 401, because the staging key was promoted to production three weeks ago and nobody rotated it — or the model id in the production config was retired, or the base URL points at a region this account cannot use. A suite built on mocks cannot see any of that, by construction.

The gap a mocked suite cannot cover

Mocks answer the question “does my code behave correctly given this response?” The deploy-time question is different: “can this deployed artefact, with this environment’s configuration, reach the provider at all?” Everything in the second question lives in the parts your tests replaced.

The specific failures a live smoke test catches, none of which is exotic:

• An API key that is missing, malformed, revoked, expired, or belongs to the wrong organisation or project.
• A key without the scope or permission for the endpoint — a 403 rather than a 401, which fails differently and is often handled worse.
• A model id that no longer exists, was never available on this account, or is not enabled in this region.
• A base URL, region or gateway hostname that is wrong for the environment — the single most common staging-to-production difference.
• Egress that is blocked: a security group, a proxy requiring credentials, a DNS entry that resolves only inside the VPC.
• A billing state that blocks requests — a spend cap reached, a payment failure. Anthropic returns a distinct 402 billing_error for this, which is worth recognising separately from an auth failure.

What it asserts, and what it must not

The discipline that makes this useful is refusing to let it grow. A smoke test that starts checking answer quality becomes slow, becomes flaky, and then gets disabled after it blocks a deploy for a reason nobody believes. Assert only:

• The call returns a success status. Not a specific body — the status and the absence of an error envelope.
• The response is structurally complete: a non-empty content block, astop_reason or finish_reason drawn from the set your code handles, and a usage object with a positive input token count.
• The model id in the response equals the one you requested. This catches a silent substitution and confirms the id resolved.
• Total time is under a deliberately generous ceiling — ten seconds, not two. This is a liveness check, not a latency regression test, which is a different job with a different statistical basis.
• One structured-output or tool-calling call, if your product depends on those, because they can be unavailable for a model that answers plain text fine.

And never assert the text. “Reply with OK” followed by expect(text).toBe("OK") is the mistake this whole territory exists to correct: the model may reply OK. or Okay or add a sentence, and none of that is a reason to block a deploy. If you want a content assertion, assert non-emptiness and a length ceiling.

The script

1. Write it as a standalone script, not a test file, so it can run in an environment that has no dev dependencies installed — a container, a post-deploy job, a bastion.
2. Read configuration from exactly the same place the application does. If the app reads a secret from a mounted file, the smoke test reads the mounted file. A smoke test with its own configuration path tests its own configuration path.
3. Cap the cost: max_tokens of 16 and a prompt of a few tokens, so a run is a fraction of a cent and cannot become a budget line.
4. Exit non-zero with a message that names the failing check and the provider request id, so the deploy log says what to do.

#!/usr/bin/env node
// scripts/smoke.mjs — run after deploy, before traffic is switched.
import { config } from "../dist/config.js";   // the app's own config loader

const MODEL = config.model;
const started = performance.now();

const res = await fetch(`${config.baseUrl}/v1/messages`, {
  method: "POST",
  headers: {
    "content-type": "application/json",
    "x-api-key": config.apiKey,
    "anthropic-version": "2023-06-01",
  },
  body: JSON.stringify({
    model: MODEL,
    max_tokens: 16,
    messages: [{ role: "user", content: "ping" }],
  }),
});

const elapsed = performance.now() - started;
const requestId = res.headers.get("request-id") ?? "none";
const body = await res.json();

function fail(check, detail) {
  console.error(`SMOKE FAIL [${check}] ${detail} (request-id=${requestId})`);
  process.exit(1);
}

if (!res.ok) fail("status", `HTTP ${res.status} ${body?.error?.type}: ${body?.error?.message}`);
if (body.model !== MODEL) fail("model", `asked for ${MODEL}, got ${body.model}`);
if (!body.content?.length) fail("content", "empty content array");
if (!["end_turn", "max_tokens", "stop_sequence", "tool_use"].includes(body.stop_reason))
  fail("stop_reason", `unrecognised: ${body.stop_reason}`);
if (!(body.usage?.input_tokens > 0)) fail("usage", "no input token count");
if (elapsed > 10_000) fail("latency", `${Math.round(elapsed)}ms exceeds the 10s liveness ceiling`);

console.log(`SMOKE OK model=${body.model} ${Math.round(elapsed)}ms request-id=${requestId}`);

Keep the whole thing to two calls: one plain completion, and one exercising whichever structured capability you depend on. The temptation is to add a third for embeddings, a fourth for a second model, a fifth for the moderation endpoint — and each addition multiplies both the run time and the probability that a deploy is blocked by something unrelated to the deploy. If a capability matters enough to check on every release, it probably deserves its own scheduled monitor rather than a place in the critical path.

Wiring it into the deploy

Position matters more than the script does. It must run against the deployed artefact, in the target environment, before traffic reaches it — after the new version is running and before the load balancer, DNS or traffic weight is switched. Run it earlier and you are testing CI’s configuration; run it later and users found the problem first.

Give it its own API key. A dedicated smoke key means its usage is visible separately in provider billing, it can be revoked without touching production, and it makes the test’s cost countable. Run it in production too, on a schedule — a smoke test that only runs on deploy days will not notice a key that expires on a Sunday.

When it fails

A failure should roll back or halt the rollout, not merely warn: the conditions it detects are total, not marginal, and a deployment that cannot reach its provider is not serving. Because the checks are named individually, the failure message tells the operator whether this is a credential problem, a configuration problem or a provider outage — three different responses, and guessing between them at 03:00 is how a five-minute rollback becomes an hour.

One nuance. If the failure is a provider outage rather than your configuration, blocking the deploy is arguably wrong — the new version is fine and the world is not. This is where a documented override earns its place, and where distinguishing a 401 from a 529 in the failure message pays for itself. For the gradual version of the same idea, where a small share of real traffic validates a change before the rest follows, see canary releases.