Is the Quantised Model You’re Calling the Same Model?

A model name in an API is a label, not a checksum. The weights, precision, kernels and serving stack behind it can change without notice. This page does not claim any provider does that — it gives you a procedure to find out for yourself, and, more importantly, the controls that stop you concluding it on noise.

The problem, stated carefully

When you call some-vendor/some-model you are trusting that the artefact serving you today is the artefact that served you last month. Several legitimate things can change it: a provider switching to FP8 to fit more users, a kernel upgrade, a different tensor-parallel degree, a silent point release of the weights, or your requests being spread across several backends with different builds.

Most of those are invisible in the text output — a quantised model still writes fluent, plausible answers. The failure mode is a slow quality drift you attribute to your prompt. So the question worth answering is not “is it quantised”, which you usually cannot determine, but the narrower and answerable “is this endpoint’s output distribution the same as the one I recorded before?”

What signal is actually available

• Log probabilities. Where an API returns logprobs, you get the model’s distribution rather than one sample from it. This is by far the strongest signal — numeric precision changes shift logits slightly and consistently, whereas text sampling hides that behind randomness.
• Greedy continuations. With temperature 0 and a fixed prompt, divergence point — the token index at which two runs first differ — is a usable proxy when logprobs are unavailable.
• Tokens-per-second and its distribution. A step change in decode rate, especially alongside a distribution change, is corroborating evidence of a different serving configuration.
• What is not signal: a single answer looking worse. A model that gets one hard question wrong tells you nothing; you are sampling from a distribution and one draw has no statistical content.

The probe

Build a fixed probe suite of 100–200 short prompts that end where the next token is meaningfully constrained but not certain — partial code lines, mid-sentence factual completions, a formatted list halfway through. Then record the top-k distribution for each, and compare against your reference recording with a symmetric divergence.

import fs from "node:fs";

type Dist = Record<string, number>;   // token -> probability

async function probe(prompt: string, model: string): Promise<Dist> {
  const res = await fetch("https://api.example.com/v1/chat/completions", {
    method: "POST",
    headers: { "content-type": "application/json", authorization: KEY },
    body: JSON.stringify({
      model,
      messages: [{ role: "user", content: prompt }],
      max_tokens: 1,
      temperature: 0,
      logprobs: true,
      top_logprobs: 20,
      seed: 7,
    }),
  });
  const j = await res.json();
  const top = j.choices[0].logprobs.content[0].top_logprobs;
  const d: Dist = {};
  for (const t of top) d[t.token] = Math.exp(t.logprob);
  return d;
}

// Jensen-Shannon divergence, in bits. Symmetric, bounded in [0, 1],
// and finite even when the two distributions have different support --
// which KL is not, and top-k lists always have different support.
function js(p: Dist, q: Dist): number {
  const keys = new Set([...Object.keys(p), ...Object.keys(q)]);
  let d = 0;
  for (const k of keys) {
    const a = p[k] ?? 0, b = q[k] ?? 0, m = (a + b) / 2;
    if (a > 0) d += 0.5 * a * Math.log2(a / m);
    if (b > 0) d += 0.5 * b * Math.log2(b / m);
  }
  return d;
}

const suite: string[] = JSON.parse(fs.readFileSync("probes.json", "utf8"));

export async function sweep(model: string) {
  const out = [];
  for (const prompt of suite) {
    // three independent calls per prompt: two for the noise floor,
    // one to compare against the stored reference.
    const [a, b] = [await probe(prompt, model), await probe(prompt, model)];
    out.push({ prompt, self: js(a, b), dist: a });
  }
  return out;
}

The important design choice is the self column. Every run measures the endpoint against itself as well as against the reference, because you need to know how much divergence the endpoint produces when nothing has changed at all.

Establishing the noise floor first

Identical requests to an unchanged endpoint do not return identical logprobs, and if you skip this step you will detect a change that did not happen. The reasons are well understood:

• Batch-dependent reductions. Floating-point addition is not associative, and the order in which a kernel reduces depends on the batch shape, which depends on other users. Same weights, same input, slightly different logits.
• Mixture-of-experts routing can depend on the composition of the batch in some implementations, which perturbs results for reasons that have nothing to do with you.
• Load balancing across heterogeneous hardware. Two requests may land on different GPU generations with different kernel selections.
• A seed parameter fixes sampling, not arithmetic. It removes one source of variation and leaves the others.

So: run the suite twice against the endpoint in one session, take the distribution of the self-divergence, and treat its high percentile as your noise floor. A real change is one where the reference divergence sits well outside that floor across many prompts at once.

Interpreting a difference

What you can conclude, and what you cannot, in order of strength:

• Sound: “The output distribution on my probe suite is different from the one recorded on this date, by an amount far outside the same-session noise floor, on most prompts.” That is a factual, reproducible statement about behaviour.
• Reasonable: “The change is consistent with a different numeric format or build.” Precision changes tend to perturb many prompts by a small amount rather than a few prompts severely, so the shape of the divergence distribution is informative.
• Not supported: “They switched to INT4.” You cannot recover a format from output. Divergence is a fingerprint, not a bit width.
• Not supported: “It got worse.” Different is not worse. If you want a quality claim you need your task evaluation, not a divergence number — but the probe is what tells you when to bother re-running the evaluation.

Run the sweep on a schedule, store the divergences as a time series, and alert on a sustained level shift rather than a single spike. The practical value is not catching anybody out; it is knowing whether a quality regression in your own product started with you or with your provider — which is a question that otherwise consumes a week.

A note on choosing probe prompts, because the suite does most of the work. Prompts whose next token is certain — the second half of a famous quotation, the closing bracket of a well-formed expression — carry no signal, because every version of the model puts nearly all the mass on the same token and the divergence is zero regardless. Prompts whose next token is nearly uniform are equally useless, because the noise swamps everything. What you want is the middle: two or three plausible continuations with meaningfully different probabilities, which is where a small change in numeric precision actually moves the ordering. Prefixes of partially-written code, a sentence up to a hedging word, and the point just before a list item are all reliable sources.

And decide in advance what a positive result would cause you to do, because the detection is only worth building if it has consequences. Reasonable ones: re-run your task evaluation before assuming a regression is yours; pin to a provider that documents its serving precision; or ask the provider directly, which is a conversation that goes considerably better with a reproducible probe and a date attached than with an impression.