Debugging With an LLM: Effective Patterns

Paste an error message and you get the most common cause of that message across everything the model has read. That is a prior, not a diagnosis, and the difference matters most precisely when the bug is interesting.

What you get when you paste the error

A stack trace names where a problem surfaced. The bug is usually elsewhere — a value became wrong earlier and travelled. The model, given only the trace, answers the question it was asked: what usually causes this? For common errors that prior is strong enough to be right often, which is the trap, because when it is wrong it is wrong confidently and it proposes a change at the surface that makes the symptom disappear.

Debugging is an inference problem, and inference needs evidence that discriminates between hypotheses. So the whole technique is: supply evidence, demand competing explanations, and pick the observation that separates them.

Hypotheses and discriminating tests

Replace “here is my error, what is wrong” with four statements and one request.

• The invariant that is violated. Not the message — the property. “Every order in shipped must have a non-null shipped_at, and 12 rows do not.”
• Last known good. A commit, a date, a deploy, a version, or “never worked”. This single fact eliminates most of the hypothesis space and is the one people most often omit.
• What changed. Between good and bad: the diff, the dependency bump, the config, the traffic pattern, the data.
• What you have already ruled out, and how. Otherwise you will be told to check it again.

Then the request that does the work: give me three hypotheses ranked by likelihood, and for each, the single cheapest observation that would be different if that hypothesis were true and the others false.

The constraint in that sentence — different under one hypothesis and not the others — is the whole method. A test that all three hypotheses predict identically tells you nothing however cheap it is, and suggesting one is the most common way a debugging session goes in circles.

Two errors where the obvious fix is wrong

TypeError: Cannot read properties of undefined (reading ‘map’)

The population’s answer is optional chaining: items?.map(...). It removes the exception and keeps the bug — now a list silently renders empty and nobody is paged. The useful framing is that items is undefined and something was supposed to set it. Three hypotheses and their discriminating observations:

• The fetch failed and the error branch left state untouched — check whether the network tab shows a non-2xx, or log in the catch.
• The component rendered before the fetch resolved and the initial state is undefined rather than [] — check the initial value; this predicts the error appears exactly once on mount and never again.
• The response shape changed and the field is now nested — log Object.keys(response); this predicts the error is deterministic and survives a retry.

Each predicts a different observation. Fifteen seconds of logging decides it, and the fix differs in all three cases.

Error: read ECONNRESET in a Node HTTP client

The prior says “add a retry”. Sometimes correct, often masking a specific and fixable race: Node’s HTTP server closes idle keep-alive connections after server.keepAliveTimeout, which defaults to 5 seconds, and a client whose own idle timeout is longer will pick a socket the server is closing at that instant.

The discriminating observation is beautifully specific: plot the inter-arrival gap before each failure. If failures cluster at requests that follow an idle period near the server’s keep-alive timeout, it is the race, and the fix is to make the client’s idle timeout shorter than the server’s — not to retry. If failures are spread across gaps uniformly, it is a genuine network or peer fault and the retry was right after all. Same error string, two causes, one cheap measurement between them.

Let a machine do the search

If you have a last-known-good commit and a reliable reproduction, do not ask a model to guess which change caused it. That is a search over commits, and git bisect performs it exactly:

# repro.sh: exit 0 if the behaviour is good, 1 if bad, 125 to skip
git bisect start HEAD v2.4.1
git bisect run ./repro.sh
# 14 revisions, ~4 steps -> "a1b3f9c is the first bad commit"

Roughly log₂(n) builds finds it among any number of commits. Then hand the model the offending commit and the repro — a well-posed question with the search already done, which is the shape it is reliable on. Writing repro.sh is itself a good delegation, and the discipline of writing one is most of the debugging anyway.

Where this stops working

Be honest about the categories where a model is weak, so you stop early rather than iterating.

• Anything involving time or concurrency. The evidence is an interleaving, which does not fit in a paste, and the model has no way to observe it.
• State outside the code. A stale cache, a migration that half-ran, a feature flag, a row with a NULL nobody expected. The model reasons about the code you showed it and the bug is in the data.
• Environment differences. Works locally, fails in CI. Paste the two environments — versions, flags, locale, timezone, container base image — or the conversation is fiction.
• Anything where you cannot reproduce it. No reproduction, no discriminating test, no method. Get one first.

There is one use that works even in those categories, and it is the oldest debugging technique there is. Explain the failing code to the model, line by line, in your own words, and ask it to tell you where your explanation stops matching what the code does. You will often find the bug in the middle of your own sentence — which is rubber-ducking, except that this duck reads the code and interrupts. It is also the one pattern where being wrong costs nothing, because you are the one doing the reasoning and it is only checking you.

One rule to end on: a fix you cannot explain is not a fix. Before merging, require a causal chain in one sentence — this input produced that state, which violated this assumption here — and a test that fails without the change. If neither the model nor you can produce the sentence, what you have is a change that made the symptom stop, and it will be back.