What We Would Do Differently, Starting Over

Written at the point where the system works, is deployed, and has barely been used — which is the most honest moment to write it, because nothing here is being reverse-engineered from an outcome we already know. Some of these will look different in a year.

The decisions worth keeping

The six we would reverse on day one

1. Test the third-party path against the third party, immediately. The auto top-up defect — an invoice built in the wrong currency, which could never have charged anybody — was found by the first run of a check against the real payment provider in test mode, and could only ever have been found there. The half we had tested was our half, which was the right half to test first and structurally incapable of seeing it. Any integration where a third party has rules of its own gets a real end-to-end check before it is called finished.
2. Ship the shared UI primitives before dispatching anyone. Including the ones you do not think you need. Three agents each wrote their own select control within minutes of each other; two hundred pages with two hundred slightly different dropdowns is exactly the look shared components exist to prevent. Converging afterwards costs far more than guessing generously up front.
3. Run one agent, review it properly, then fan out. Twenty were dispatched on a detailed brief and it worked, with luck. A single reference pass would have caught the missing primitives and a test whose file pattern silently skipped twenty finished pages, before either was multiplied by twenty.
4. Cost the architecture’s bundle on day one. One eager registry importing every client component put 441 kB of first-load JavaScript on the pages whose entire purpose is to rank. The seam to fix it existed from the beginning and went unused because nothing forced the question until the final build. Measure the number you care about while the change is still cheap.
5. Register every test suite the day it is written. The strongest suite in the repository — the one that deletes the catalogue, routes every balance change through the auditable path and reconciles the ledger as its last assertion — had no script entry, no documentation and no continuous integration. It ran when somebody remembered the path. A suite nobody can invoke is a suite that does not run.
6. Make prices and model identifiers data with provenance from the start. Each price carrying the page it came from and the date it was read. A price embedded in a calculation fails silently and self-consistently: the ledger reconciles perfectly against an out-of-date figure while you systematically mis-charge, and every audit stays green.

The order we would build in

Not a general prescription — a specific claim about a metered product with an unreliable, chargeable dependency.

1. The ledger and the reconciliation audit. Before any feature. It is the thing that tells you whether everything after it is working.
2. One real end-to-end path against the real third party — one provider, one payment, one webhook — before breadth. This is the step we would move earliest and it is the one that found the most expensive defect here.
3. The per-attempt request record. Provider, route, classification, cost, attempt index. Every diagnosis afterwards depends on it, and it is the thing that is painful to retrofit.
4. The abstraction boundary — one adapter per provider, one internal shape, unsupported parameters refused rather than forwarded — before the second provider, not after the third.
5. The off-switches, all four layers, before the first feature that spends money. See designing the off-switch.
6. Then features. Everything above is two weeks and makes every subsequent week cheaper.

What was cheap, and what was not

Estimates were wrong in both directions, and the pattern is consistent enough to be worth stating: everything with a specification was cheap, and everything at a boundary with somebody else was expensive.

Cheaper than expected

• Volume of writing. A thousand pages was the thing that looked impossible and was solved in two working sessions. The constraint was never throughput; it was the rules that keep a thousand pages from being a liability.
• Breadth of provider support. Adding an adapter behind an existing internal shape is a day. It only stays a day if the shape was designed before the second provider, which is the argument for doing it early.
• Auditing our own code. Reading a lane and comparing sibling paths — six of them have a permission check and the seventh does not — is mechanical, and mechanical work is cheap now. This produced most of the findings worth having.

More expensive than expected

• Anything with a third party’s rules in it. The payment integration, tax behaviour, and how each provider signals an exhausted balance. None of this is difficult; all of it is undiscoverable except by running it against the real service, and the gap between “the code looks right” and “the third party accepted it” was where the worst defect in the project lived.
• Getting the same fact to agree in two places. The schema and the database, the registry and the files, the catalogue and the price list, the suite and the runner. Every one of these joins broke at least once and each side looked complete on its own.
• The last 10% of anything user-visible. Empty states, loading states, error states, and what a feature looks like when it is switched off. These are the parts nobody specifies, and they are disproportionately what a first user actually encounters.

Two we still cannot call

Both of these will read as obvious in a year, in one direction or the other, and pretending to know now would be the same failure this whole cluster exists to avoid.

• Whether a thousand pages was the right investment. The library is days old. Nothing has ranked. The production problem was solved and the demand question is entirely untested, and any confident statement about it right now would be invented.
• Whether publishing all of it at once was expensive. The staggered queue existed precisely because a bulk release from a four-day-old domain is the strongest version of a pattern search engines treat as a signal. It was overridden deliberately, with the cost stated. The mechanism to return to a cadence survives intact, which is the part we would keep either way.

If you take one thing

Put the assertions on the seams. Almost everything that broke here broke where two representations of the same fact were supposed to agree and nothing checked that they did: the schema and the database, the registry and the files, the comment and the code, the exclusion list and the archive, the price list and the invoice, the suite and the runner.

The behaviour itself was mostly fine. It was the joins that were wrong, and joins are exactly what nobody writes a test for — because each side looks complete on its own. That is the whole of it, and it is set out concretely in what our evals missed and things we believed that were wrong.