What Our Evals Missed

Every defect below existed while the suites were green. None of them was a missing assertion in the ordinary sense — each was invisible for a structural reason, and the structural reason is the part worth stealing.

A green suite is a claim about the suite

The gateway this is written about has a substantial test estate: an end-to-end suite of 142 assertions that gates deploys, a money-paths suite of over 200 that routes every balance change through one function so a ledger audit means something, plus unit suites for cryptography, money parsing, guardrails, two-factor codes and upstream billing.

All of it was passing when each of the following was true. That is not an argument against testing; it is an argument that the interesting question is never “does it pass” but “what could it not have seen”.

The suite that was buying real inference

The deploy-gating suite had one assertion that failed roughly once in three runs and passed otherwise. It was recorded in the README as flaky and not diagnosed. The diagnosis turned out to be much worse than the flake.

The suite wipes a throwaway copy of the database before running, and the wipe deliberately kept the seeded catalogue. That catalogue holds six real providers and 329 real routes. The application under test is spawned with the whole environment, so a real provider key reaches it, and every one of those routes therefore looks reachable.

One assertion exercised the automatic router, which picks the cheapest model that fits the request. The cheapest rows in the real catalogue are aggregator routes priced at a fraction of the mock provider’s rate — so the router resolved to a real vendor model and made a real, billed HTTP call, on our credit, on every run. The assertion passed or failed with a third party’s availability. That was the flake.

The mitigation already existed. The money-paths suite deletes the real catalogue and explains why in one sentence that the other file needed: a real provider row in a test database is one routing mistake away from a real invoice. It had never been carried across. The fix was to delete the catalogue there too, and to rewrite two assertions that had been reading production seed data — a count of more than thirty models and more than five providers, thresholds only the real catalogue could clear, which is exactly what made deleting it look like it would break the suite.

Three suites nothing could run

The strongest test asset in the repository — the money-paths suite, which deletes the catalogue, routes every balance change through one auditable function and runs a full ledger reconciliation as its last assertion — had no npm script, no README mention and no CI entry. Neither did the end-to-end suite that gates deploys, nor two unit suites.

They ran when somebody remembered the path. Nothing enumerated them, so nothing noticed when they were not run, and a new contributor listing the available scripts would have concluded they did not exist. Writing a strong suite and leaving it unregistered is functionally the same as not writing it; the fix was four lines of configuration.

The same failure mode had a sibling in the content library: a test that discovered cluster files by matching a filename pattern silently skipped twenty pages, because one file was named with a suffix the pattern did not match. No route broke. No test failed. The pages simply were not there, and an agent reading the test found it.

Both are instances of one assertion that is worth having in any project that generates files: assert that the filesystem and the registry agree. Every module on disk is enumerated somewhere, and every entry in the enumeration exists on disk.

Everything was sequential, so races were untested

Every suite in the repository ran its assertions one after another. The invariants that only break under parallelism — which, for a product that holds customer balances, are the whole business — had never been exercised at all.

A concurrency suite was written specifically to attack them: eight simultaneous requests against credit for three, one idempotency key used by six clients at once, a per-key cap and a monthly ceiling raced against each other, twelve parallel top-up claims against a daily ceiling, a cache stampede, concurrent streams. It re-checks the ledger identity after every phase rather than only at the end.

It found a defect on its first run and deliberately did not fix it. The reservation function tests balance, key cap and monthly ceiling in a single conditional and returns the same null for all three, so a refusal that loses a race is reported as insufficient credit whatever actually refused — a customer at their own key cap, on an account with plenty of credit, is told to top up. A refusal type existed in the source and nothing read it. It was pinned as a known bug so the suite stays a regression net and the defect stays loud. It is a wrong message, not a wrong charge, which is why it was recorded rather than rushed.

Four defects found by running, not reading

Four defects in shipped interactive tools were found by executing them against a reference, not by review:

• An approximate-nearest-neighbour demo with 27% recall. Its insertion loop started the search at the new node’s own level rather than at the graph entry point, so the bottom layer split into four disconnected islands and recall was flat regardless of the search-effort parameter. After the fix, recall moved from 78% to 96% across the same parameter sweep. This is the frightening one: 27% is a plausible number that would have read as a lesson about approximate search rather than as a bug.
• A PDF reader that failed on every compressed file, because the newline the specification requires before a stream terminator is not part of the stream data, and the decompressor rejected it.
• A schema emitter that wrote nested classes after the class referencing them — valid-looking output that will not import.
• A markdown parser that treated a heading inside a fenced code block as a new document.

What caught them was differential testing against an independent reference: hashing checked against the published standard vectors including the million-character case, a hand-written regular-expression engine checked against the platform’s own across hundreds of combinations, a diff algorithm checked against a brute-force longest-common-subsequence on 500 cases. A reference implementation you did not write is worth more than any number of assertions you did.

When the test is wrong and the code is right

Two assertions added alongside a bundle-size refactor failed on their first run, and both faults were in the tests. One was checking a property against metadata that no longer carried it; the other built its expectation per-item, so two legitimately empty modules looked like references pointing at nothing.

The same thing happened in a Stripe integration check: an assertion about a tax field was wrong rather than the code, because the field is not echoed where the test expected it. What replaced it was the invariant that holds either way — total equals subtotal plus tax — which is a better assertion than the one it replaced.

A failing new test is not automatically a found bug, and the discipline that matters is to determine which of the two is wrong before changing either. A test rewritten to match wrong behaviour is worse than no test, because it now certifies the defect.

The pattern underneath all of them

Every blind spot here is the same shape: the suite was testing the thing and not the seam.

If you take one thing: the highest-value assertions in a large project are almost never about behaviour. They are about two representations of the same fact agreeing with each other.