Updating Uptime Monitoring for a New Model Provider

The synthetic check survived the migration without being edited, and it is still green. That is not reassurance. A check that asserts on a JSON path which no longer exists usually asserts on nothing at all.

The check that passes and tests nothing

Most LLM uptime checks are one of two shapes, and both fail the same way at a cutover.

The first hits a cheap metadata endpoint — a models listing, an account endpoint — and asserts HTTP 200. This never tested inference in the first place. It tells you the control plane is up, which is a different system from the one serving your tokens, and it will stay green through an inference outage.

The second sends a real completion and asserts something about the body. This is the right idea, and it is the one that quietly rots. A check written against Chat Completions asserts on choices[0].message.content. The new provider returns text under a different path — on the Messages API, a content block array where the first block carries a text field. If the assertion was written as “response body contains non-empty text” rather than as a typed extraction, the whole JSON document satisfies it: the model identifier alone is non-empty text. The check now passes on any 200, including a 200 that returned an empty completion.

The tell is that the check never failed during the migration itself. If you swapped providers and the synthetic monitor did not go red for even a minute, it was not asserting on anything provider-specific, which means it was not asserting on anything.

What a useful probe asserts

Five assertions, in order of how often each catches something real.

• A deterministic answer. Ask something with exactly one right answer at temperature 0 — a fixed arithmetic question, an extraction from a fixed short document — and assert on the value after extracting it from the correct path. This catches the empty completion, the wrong model being served, and the credentials pointing at the wrong account.
• The terminal reason. Assert that generation ended normally: stop_reason equal to end_turn, or finish_reason equal to stop. A probe returning length or content_filter is a probe whose answer assertion is about to become flaky for reasons unrelated to availability.
• Latency, split. Record time to first byte separately from total. On a streaming probe these are different signals: a healthy time to first token with a slow total means degraded throughput, and a slow first token means queueing. Collapsing them into one number loses the distinction you need at 03:00.
• The rate-limit headers exist and are sane. Anthropic documents anthropic-ratelimit-requests-remaining, anthropic-ratelimit-input-tokens-remaining and anthropic-ratelimit-output-tokens-remaining, with matching -limit and -reset headers, on its rate limits page. Recording the remaining values from the probe gives you a free headroom gauge; a remaining count trending to zero is a warning you would otherwise get as a page.
• Token accounting is present. Assert the usage object exists and its counts are non-zero. This is the assertion that catches a provider returning a cached or stubbed response.

Building the probe

1. Pin the model string explicitly, never an alias that resolves to “latest”. A probe against a moving alias tells you about availability and about model changes at the same time, and you cannot tell which fired.
2. Set temperature to 0 and the output cap to something small — tens of tokens. The probe is testing that the service answers, not that it writes well.
3. Extract the answer through a typed adapter, one function per provider, that returns null when the shape is not what it expects. The null case is a failure, not an empty string. This is the single change that prevents the vacuous-check problem recurring.
4. Assert on the extracted value, on the terminal reason, and on the usage counts. Emit latency and remaining-quota as metrics rather than assertions, with alerting thresholds set separately.
5. Run the same probe against every provider you can fail over to, not just the one in production. A fallback path nobody probes is a fallback path that is discovered broken during the incident it was meant to cover — the point of testing fallback order.
6. Run from more than one region if you serve more than one, because regional endpoints fail independently. The endpoint naming differs by provider; see regional endpoint structure.

// probes/inference.ts — one adapter per provider, null on unexpected shape
type Probe = { text: string | null; ended: string | null; inTok: number; outTok: number };

export function readMessagesApi(body: any): Probe {
  const block = Array.isArray(body?.content)
    ? body.content.find((b: any) => b?.type === "text")
    : null;
  return {
    text: typeof block?.text === "string" ? block.text : null,
    ended: body?.stop_reason ?? null,
    inTok: body?.usage?.input_tokens ?? 0,
    outTok: body?.usage?.output_tokens ?? 0,
  };
}

export function readChatCompletions(body: any): Probe {
  const msg = body?.choices?.[0]?.message;
  return {
    text: typeof msg?.content === "string" ? msg.content : null,
    ended: body?.choices?.[0]?.finish_reason ?? null,
    inTok: body?.usage?.prompt_tokens ?? 0,
    outTok: body?.usage?.completion_tokens ?? 0,
  };
}

// The assertion that does not rot:
//   probe.text?.trim() === EXPECTED && probe.ended === OK_REASON && probe.outTok > 0

Classifying the failure so the page is useful

A probe that reports “down” without saying which kind of down wakes the wrong person. Split the outcome into four categories at the probe, because the response differs for each: an authentication or permission failure is a configuration problem for whoever rotated the key; a rate-limit or quota failure is a capacity problem and may be self- inflicted; a server or overload failure is the provider’s and the action is failover; a shape failure — 200 with an unparseable body — is usually your own adapter and a deploy is the likely cause.

The status codes that carry those categories are not the same everywhere, which is the subject of the error-recovery page, and the probe should reuse the same normalisation rather than re-implementing it. If the probe and the production client disagree about what counts as retryable, the probe is not testing production.

Keeping the probe cheap

A probe at one-minute resolution against three providers from three regions is roughly thirteen thousand paid calls a day. Small ones, but not free, and they land in the same billing bucket as production traffic, which will confuse the cost dashboard unless you tag them. Send an identifying header or user field, exclude it in the cost view, and put a line for it in the cost table so nobody investigates it as an anomaly.