What You Lose Moving From a Managed Threads API to Your Own Loop

Teams leaving a managed threads API usually budget for the database. The database is the cheap part. What the abstraction was really providing was a set of policy decisions nobody had to make, and those arrive all at once on the day you leave.

The bargain the abstraction offers

A managed threads-and-runs API is not primarily a storage service, although that is how it is described and how it is priced. It is a bundle of defaults. It decides how a conversation is ordered, what happens when the conversation outgrows the context window, how long a tool loop may run before it is abandoned, what a partially-failed run leaves behind, how retrieval is chunked, and how long any of it is kept. Every one of those is a real decision with a real wrong answer, and the API ships an answer to all of them.

That is why it feels so much lighter than it is. The value is proportional to the number of decisions you did not have to make, which is exactly the quantity that is invisible while things work. The conversion at the exit is unfavourable in the same way: you are not replacing a feature, you are replacing a default, and a default has no specification to port.

It is worth being clear that this is not an argument against leaving. The pages in this cluster spend most of their length on how to leave, because for a great many applications leaving is correct. It is an argument about what to put in the estimate.

What you inherit at the door

• Ordering, exactly. Messages must be replayed in the order they occurred, and a tool result must follow the assistant turn that requested it. Timestamps are not sufficient — concurrent writes collide — so you own a sequence, and with it the question of what happens when two requests arrive on one conversation at once. The managed API serialised runs per thread; nothing serialises yours until you write the lock.
• Termination. A run had a lifecycle: it could expire, it could be cancelled, and it could not loop forever. Your loop has whatever ceiling you gave it, and if you gave it none, a model that keeps requesting tools will spend your budget until something else stops it (a step budget is testable).
• Idempotency at the tool boundary. Managed or not, a retried tool call can fire a side effect twice — but the managed run’s at-most-once submission window made this rarer by accident. Once you own the loop, retries are yours and the guarantee has to be built (no duplicate side effects on retry).
• Partial failure. A run that failed halfway left a recorded state you could inspect. Your loop, if it throws in the middle, leaves a conversation whose last assistant turn requested tools with no results appended — which is not merely untidy, it is a state the next request will reject. Recovering from your own half-finished turns is code, and it is code nobody writes on the first pass.
• Retention and deletion. Server-side threads came with a retention story and a delete endpoint that satisfied a data-subject request in one call. Now conversation history is in your database, inside your backups, inside your replicas, and deletion means deleting it from all of those. This is the one that arrives as a compliance question months later rather than as an engineering question on day one.
• Observability of the middle. Run steps gave you a record of what happened inside a run without your having instrumented anything. Your loop emits what you tell it to, and the interesting failures are the ones in the middle — what to log stops being a nice-to-have the moment the loop is yours.

The one that surprises people

Of that list, truncation policy is the item most consistently underestimated, and it deserves the argument spelled out.

A managed thread is unbounded. A context window is not. Something must decide what to drop, and while the thread was managed that something was a documented, configurable strategy with a default — you could leave it alone for the entire life of the product and never think about it. The day you own the loop, you own a policy that silently determines what your model can remember, and it has no obviously correct value.

Keeping the last N turns loses the constraint the user stated at the start. Summarising the old turns loses fidelity in a way that is impossible to notice from the outside, because the model answers confidently from a summary that dropped the important clause. Retrieving over the conversation reintroduces a retrieval pipeline you left to avoid. All three are defensible and all three fail differently, and the failure is never an error — it is an answer that ignores something said forty messages ago (multi-turn context loss).

The reason this bites late is that it is invisible in testing. Test conversations are short. Truncation only engages on long ones, so the policy is exercised for the first time by your heaviest real users, weeks after launch, and it presents as vague complaints about the assistant getting worse over a long session rather than as anything a dashboard shows. If you take one thing from this page into the migration plan, make it a fixture set of deliberately over-long conversations, and make it before the cutover rather than after the complaints.

What you actually gain

The gains are real, and they are the mirror image of the losses: every decision you inherit is also a decision you can now make correctly for your case.

Model portability is the largest of them. A conversation stored as provider-neutral rows can be replayed against any provider, which means a version bump becomes a config change rather than a migration and a provider outage becomes a fallback rather than an incident. That is impossible while your conversation state lives inside one provider’s object model — the state is the lock-in, not the API shape. Provider-agnostic code is only meaningful once the data is portable too.

Then: prompt versioning that works, because you control exactly what is sent; caching, because you control the prefix; cost attribution per conversation, because the usage figures land in your own rows; truncation tuned to your content rather than to a general default; and the ability to run an evaluation over historical conversations, which requires having them in a queryable store. Several of these are not improvements on the managed version — they were simply not available at all.

When leaving is the wrong call

The honest cases against, since an argument that never concedes one is not an argument.

If the assistant leans on managed retrieval and your team has not built a retrieval pipeline before, the migration is not the loop — it is a RAG project with a deadline attached, and it will be estimated as the loop. If the product is early enough that conversation shape is still changing weekly, owning the schema means migrating it weekly. If the deployment has no operational surface to speak of — no on-call, no backup story, no data-deletion process — then inheriting retention and partial-failure recovery is inheriting obligations there is nobody to discharge.

And the timing case: a provider deprecating a managed API is a reason to move, but moving to that provider’s replacement managed API is a smaller move than moving to your own loop, and it is legitimate to take the smaller one now and the larger one when it buys something concrete. The argument here is not that managed abstractions are bad. It is that they are priced in decisions rather than in dollars, and the invoice for those arrives on the day you leave, in full, at once.