Nothing Changed and the Output Changed

“Nothing changed and the output changed” has three possible verdicts: it did not actually change, you changed something, or they changed something. Work through them in that order, because the first two are cheap to rule out and the third is the one that needs evidence you may not have collected yet.

Establish that it changed at all

Start with the evidence you have. A single screenshot and a memory of how it used to behave is not enough to investigate against, and half of these investigations end here.

1. Get the exact inputs and outputs, before and after. If you log prompts and responses, pull the specific request IDs. If you do not, that is the first fix and it is the reason the rest of this page is hard.
2. Establish when. Query for a measurable proxy — mean output length, refusal rate, JSON parse failure rate, a validator’s pass rate — by day. A step change on one date is an event to correlate against. A gradual drift is a different investigation, usually about your inputs.

   SELECT date_trunc('day', ts) AS day,
          count(*)                                   AS n,
          avg(completion_tokens)                     AS out_len,
          avg((status = 200 AND parse_ok)::int)      AS parse_rate,
          avg(refused::int)                          AS refusal_rate
   FROM llm_requests
   WHERE route = 'the-affected-one' AND ts > now() - interval '30 days'
   GROUP BY 1 ORDER BY 1;

3. Reproduce it now. Take an exact input from before the change and send it again, unmodified. If the output is now different, you have a live reproduction and everything below is fast. If it is the same, the change is in your inputs, not in the model.

Rule out chance, which is free

At any temperature above 0 the output is a sample. Behaviour you think of as “how it works” may have been an 80% behaviour all along, and three consecutive draws from the other 20% look exactly like a regression.

from collections import Counter

def rate(prompt, check, n=30, **kw):
    hits = Counter()
    for _ in range(n):
        out = call(prompt, **kw)
        hits[bool(check(out))] += 1
    return hits[True] / n

print("now:", rate(PROMPT, CHECK))   # compare against your stored history

Thirty samples distinguishes 95% from 70%; three samples distinguishes nothing. If the rate now is 78% and you have no record of what it was before, the honest conclusion is that you cannot tell, and the fix is to start recording it. That is not a failed investigation — it is the correct answer, and it prevents a week spent chasing a change that did not happen.

Rule out yourself, which is nearly free

“Nothing changed” usually means “I did not deploy application code”, which is a much narrower claim than it sounds. Check each of these against the date from step 1.

1. Code, including dependencies.

   git log --since="2026-07-25" --until="2026-07-29" --oneline -- \
       src/prompts src/llm requirements.txt package-lock.json

   An SDK minor version can change a client-side default, a retry policy, or how messages are serialised. Lockfile diffs count as changes.
2. Configuration and feature flags. Environment variables, remote config, a flag rolled from 10% to 100%, a per-tenant override. These change behaviour with no deploy by design, which is exactly why they are easy to forget.
3. Data the prompt reads. A retrieved document was edited. A database row changed. A template pulled a value that moved. A date in the system prompt crossed a boundary. If your prompt embeds anything dynamic, it changed even though your code did not.
4. The retrieval index. A reindex, a new document set, a changed chunker or embedding model. Different context is a different prompt.
5. Input distribution. A new client, a new locale, a marketing campaign bringing a different kind of user. The model is answering different questions. Check the length and language distribution of inputs across the boundary date.

The strongest single test here: reconstruct the exact request from a logged example on the old side and replay it today. If the replayed request produces the old output, nothing about the provider changed and the difference is in what you are now sending.

Rule out them, which needs one logged field

The response body contains the model that actually served the request. For a pinned snapshot it equals what you asked for; for an alias it is the resolved version, and that is where a silent change shows up.

SELECT date_trunc('day', ts) AS day,
       model,                       -- from the RESPONSE, not the request
       count(*)
FROM llm_requests
WHERE ts > now() - interval '30 days'
GROUP BY 1, 2 ORDER BY 1, 3 DESC;

A new value appearing on the boundary date is a complete answer, and the investigation is over in one query. If you are not logging that field, log it now: it is one column and it is the difference between this being a five-minute question and a five-day one.

• An alias moved. The commonest cause in this category and entirely legitimate — you asked for the current version and got it.
• A serving change under an unchanged name. Quantisation, hardware, a serving-stack upgrade. Not always announced and not always visible in any field, which is why the statistical evidence from step 1 matters when the model name is unchanged.
• Your own routing. If a gateway or failover sent traffic to a different provider or a different deployment, that is a change on your side that looks like a change on theirs. Log the route as well as the model.
• A policy or filter update. Refusal rates can move without any model change. If the difference is specifically that requests are now declined, content filters is the page.

Where a provider exposes a system fingerprint or equivalent backend-identity field, log that too. It changes when the serving configuration changes even if the model name does not, which is the only signal available for the second bullet above.

What each verdict means you should do

Making the next one a five-minute question

1. Log the resolved model from every response. One column, and it answers this class of question outright.
2. Log a hash of the prompt and of the parameters. Two more columns that turn “did the prompt change” into a query instead of an archaeology exercise.
3. Keep a golden set and run it on a schedule. Thirty real cases with expected properties, run daily against production configuration. This detects a change before a user reports it, which is the entire difference between a maintenance task and an incident — regression testing covers the shape.
4. Record rates, not anecdotes. Parse success, refusal rate, mean output length, validator pass rate. Cheap to compute and they make step 1 instant.
5. Pin model versions and treat a version bump as a deploy. With an evaluation run before it lands. The deprecation surprise is covered in model not found, and the general problem in silent model updates and model degradation.