Things We Believed in Month One That Were Wrong

Every belief below was held confidently, was wrong, and was disproved by a specific defect in this repository. They are grouped by what the belief was about rather than by when it fell, because the pattern only becomes visible when they are next to each other.

1. That a comment describes the code beneath it

This one cost the most, repeatedly, and it is the one we would most want somebody else to take seriously.

• A comment on the cancellation path stated the invariant plainly: aborting a stream is not a way to get free inference. The code did not implement it. Completion tokens were only ever read from a terminal usage message that never arrives on an abort, so a client could stream a large answer, disconnect, and be charged a fraction of a cent for tokens the provider billed us for in full.
• A comment on the output-guardrail block path asserted that none of those paths charged anything. That path debited for a completion the provider had really generated, returned an error, and then released the idempotency claim — so a client retrying with the same key bought a second completion at full price, as often as it retried.
• A comment justified leaving a known database inconsistency alone on the grounds that foreign-key enforcement was on, so a delete would raise rather than quietly null a column. Enforcement was off. A false premise was holding up a decision not to fix something.
• A comment justified reserving funds against the first routing candidate on the grounds that a failover target is cheaper or comparable. Under the ascending price sort the product documents, the first candidate is the cheapest and every fallback costs more.

What replaced the belief: read a comment as a claim to be checked, and treat a comment that turns out to be wrong as a finding in its own right — because something was decided on the strength of it. In all four cases above, the comment was not merely stale; it was load-bearing for a decision not to look further.

2. That a compatible API means a compatible integration

Several providers advertise an OpenAI-compatible endpoint. That claim covers the request and response envelope and very little else, and the gaps are where the money is.

Providers disagree about which status code carries an exhausted account balance. Some drop unsupported parameters silently rather than refusing them, returning a fully-billed success with a missing field. Strict clones reject unknown fields that a lenient one accepts, so the same request body works on one and 400s on another. Usage totals arrive in different places, which is what made the cancellation billing bug possible.

The same belief in a different costume: that two ways of doing the same thing with one third party are equivalent. Manual top-up worked in production for weeks; automatic top-up could never have worked for anybody, because an invoice takes its currency from the customer record while a checkout session takes it from the first line item. The full write-up is in one incident, written up properly.

3. That a passing suite means the path works

The suite that gates deploys had one assertion failing about once in three runs. It was recorded as flaky and not diagnosed. The diagnosis was that the suite kept the seeded catalogue of real providers and real routes, was spawned with a real provider key in its environment, and the automatic router therefore resolved to the cheapest real model available and made a real, billed call on every run. The assertion passed or failed with a third party’s availability.

A neighbouring suite already deleted the real catalogue and said why in one sentence: a real provider row in a test database is one routing mistake away from a real invoice. The mitigation existed and had never been carried across.

What replaced the belief: a flaky assertion is information about coupling, not about luck, and it deserves more attention than its severity suggests. More in what our evals missed.

4. That declaring a constraint enforces it

The schema declared 43 foreign keys, twenty of them cascading. The database defaults enforcement off and nothing set it. Every one of those declarations was decoration: deleting an account left its ledger entries, requests, payments, memberships, keys and webhooks behind as orphans, and a row referencing a parent that did not exist inserted without complaint.

The same class appeared twice more. A deploy script excluded secrets by listing two exact filenames, and a third secrets file sitting between them on disk matched neither. A version-control ignore rule used a dash where the real filenames used a dot, so every hand-made database copy was untracked and one command away from being committed permanently.

What replaced the belief: a declaration is a statement of intent and an assertion is a mechanism. Where the difference matters, test the mechanism — the deploy now greps its own finished archive and refuses to upload if anything matching a secrets pattern survived.

5. That our own machine is a closed blast radius

“It is our own server” was doing more work than it could bear. The host also runs an unrelated production service. The upload archive contained an encryption key and five live provider keys with no restrictive permissions. Rollback archives were complete copies of the environment file, world-readable. Nothing recorded which commit the server was running, so the box was executing code that existed in exactly one place: an uncommitted file on one laptop.

What replaced the belief: the boundary is the machine, not the organisation, and version control is not the only way a secret leaves a laptop. The ignore rule had done its job perfectly and was irrelevant to the leak that mattered.

6. That a standard library function does what its name says

A money parser did this:

input.replace(/[^0-9.,-]/g, "").replace(",", ".")

The second call passes a string rather than a pattern, and string replacement in JavaScript replaces the first match only. So "1,000" became "1.000" and parsed as one dollar, and "1,234.56" became "1.234.56", where parsing stops at the second dot and returns 1.23.

It was silent in both directions, because it parsed. Nothing threw and no validation fired. A customer typing a thousand-dollar top-up saw the quote update to one dollar and had no reason to read that as anything but their own typing error. Eleven call sites were affected, including every spending-limit setter — where the same slip turns a large ceiling into a one-dollar ceiling and starts refusing the customer’s own traffic.

The replacement is not a global flag, because a comma is a decimal mark in half of Europe and a thousands separator in the US, and this is a dollar-denominated product sold from Amsterdam, so both forms arrive. Guessing wrong is a factor of a hundred in one direction and a thousand in the other, so the separator is now decided by rules rather than assumed, and the test file is the specification — 47 assertions, every one a string somebody could plausibly type.

What replaced the belief: on any code path handling money, the test file is the specification and the implementation is the commentary. See representing money in integers.

7. That writing was going to be the hard part

With a plan for a thousand pages and four written, the obvious constraint was writing throughput. It was not. Production was solved in two sessions with parallel agents; what took the thinking was everything around it — the release cadence, the cross-links to pages that did not exist yet, the registry that had to agree with the filesystem, and the rule that no page may claim a measurement nobody performed.

The corollary is uncomfortable and worth stating: since volume is no longer the constraint, volume is no longer a differentiator either. The only defensible position is the constraint we imposed on ourselves, which is that nothing on the site claims something that did not happen. This cluster is the sharpest test of that, which is why twelve of its twenty planned first-person essays were converted into engineering guides rather than written as stories.

The pattern in all seven

Six of the seven are the same mistake: a representation of the truth was trusted in place of the truth. A comment instead of the code. A compatibility claim instead of the behaviour. A green suite instead of the path. A schema declaration instead of the enforcement. An exclusion list instead of the archive’s contents. A function name instead of its semantics.

That suggests where to look first in any system of this kind: not for the hard problems, but for the places where something states a fact that nothing checks. The corresponding practice — assert that two representations of the same fact agree — turned out to be the highest-value class of test in the whole repository, and it is set out in what our evals missed.