Mapping Conversation State When an API Keeps It For You

“The API remembers the conversation” sounds like a saving of bandwidth. It is really a transfer of ownership, and the things it takes away from you are all things you did to the transcript because you had it: edit it, redact it, count it, replay it, take it with you.

Two designs for the same conversation

In the stateless design the model has no memory whatsoever between calls. Your database holds the transcript, and every request resends all of it. Anthropic’s Messages API works this way, as does the chat-completions shape: the messages array you send is the conversation, and if you send a different one the conversation is different, with no argument from the server.

In the stateful design the provider stores each response and gives you a handle to it. On the responses shape, store controls whether a response is retained and previous_response_id — the identifier of the preceding response — is how the next call continues from it. You send only the new turn. The server assembles the context.

Both are still a single stateless HTTP request from the model’s point of view: the runtime is handed a full token sequence either way, and the model has no memory in either design. The only question is who assembles that sequence. That framing is worth keeping, because it tells you that nothing about model behaviour changes — only about who holds the data and who can therefore act on it.

What the client stops storing

The concrete shape of the migration is that a row holding an array becomes a row holding a string.

-- stateless: you own the transcript
conversations(id, user_id, messages jsonb, updated_at)

-- stateful: you own a pointer
conversations(id, user_id, last_response_id text, updated_at)

That looks like a simplification and it is, right up to the first time somebody needs the transcript for a reason other than making the next call. Support wants to see what the user was told. Legal wants a deletion. An evaluation harness wants yesterday’s conversations as test cases. A cost review wants to know how large contexts were getting. In the stateless design all four are queries against a table you own. In the stateful design each becomes an API call against somebody else’s store, subject to their rate limits, their retention window and their availability.

There is also a new failure mode with no counterpart: a dangling pointer. If the stored response expires, is deleted, or belongs to an account or project you have rotated away from, your conversation row points at nothing. The stateless design cannot produce this, because the data and the pointer are the same object.

What you can no longer do

• Rewrite history. Trimming an old turn, replacing a stale system instruction, or dropping a tool result that turned out to be wrong are all one-line edits to an array you own. Against server-held state, the transcript up to a given response is fixed. The operation you have instead is branching: continue from an earlier response identifier and re-send the turns you do want, which is a different and more expensive operation.
• Control truncation yourself. When you assemble the context you decide what falls out of it at the window boundary — summarise the oldest turns, drop tool results, keep the system prompt. When the server assembles it, that policy is a parameter with a small number of settings, and “drop the middle automatically” is not the same as your policy. See behaviour at the context window edge for what you are handing over.
• Count tokens before sending. You cannot tokenise a transcript you did not assemble. Pre-flight estimation, per-tenant context budgets and “this conversation is getting expensive” warnings all become after-the-fact reads of usage fields rather than checks you can make before spending anything.
• Replay a conversation against another model. The most consequential loss during a migration specifically. Testing the same conversations against a second provider requires the transcript in a portable form; a pointer into provider A’s store is not that. This is why dual-running is markedly harder from a stateful baseline, and it is the argument the migration runbook depends on.
• Reason about retries as pure functions. A stateless request is a function of data you hold, so re-sending it is re-computing. Once the server holds state, a retry can append a turn twice — which is the whole subject of retry logic when the API holds state.

Retention, redaction and exit

Server-side state is stored user content by definition, so the engineering decision is also a data decision and the two should not be made in different rooms.

Redaction becomes asynchronous. When you hold the transcript, honouring a deletion request is a statement in your own database, effective immediately and provable. When the provider holds it, you issue a delete against their API and rely on their retention behaviour. Anything you would previously have scrubbed before storage — card numbers a user pasted, an identifier from an upstream system — has already been stored by the time you notice, because storing was the side effect of asking a question. If a flag controls whether a request is retained, treat it as a data-protection control and set it explicitly on every call rather than inheriting a default.

Portability becomes an export project. At exit, conversations you own move by copying a table. Conversations the provider owns move only through whatever read API exists, over however many objects you have, within whatever retention window has already expired some of them. That is a concrete instance of the clause type discussed in data portability, and the question worth asking before you adopt server-side state is the boring one: what does bulk retrieval of every stored object look like, and how long does it take at our volume?

The hybrid most teams end up with

The stable position is to keep your own transcript as the source of truth and treat the server-side handle as a cache. You write every turn to your database as you always did; you also record the response identifier; you continue from the identifier when you have one, and rebuild the full context from your own store when you do not.

That gives you the bandwidth and latency benefit on the common path while keeping every capability in the list above: you can still branch, still redact, still count, still replay against a second provider, still export. The cost is that you are storing the transcript twice and paying for both, which is a real cost and a small one next to discovering at exit that the only copy of your conversation history is somewhere you are leaving.

If you take one rule from this page: a pointer is not a backup, and the moment your conversation history exists only as identifiers into somebody else’s system, your migration options are set by their read API rather than by your engineering.