Migrating Off a Framework Without a Rewrite

Removing LangChain — or any AI framework — from a working application does not require a rewrite, and the rewrite is the version that fails. The framework does four separable things; each has a seam; and each can be cut in a merge of its own while the application keeps shipping. Done in the right order, the whole job is a fortnight of small changes rather than a branch that lives for a quarter.

Decide whether to do it at all

Migration is a cost with no user-visible benefit, so it needs a reason that survives being said out loud. Three good ones and two bad ones.

Good: you are pinned to an old version because upgrading breaks something you cannot diagnose, and security updates are arriving upstream. You are regularly reading framework source to find out what prompt was sent. Or a dependency conflict between two integration packages is blocking an unrelated upgrade. All three are the framework costing you engineering time on a recurring basis.

Bad: a blog post said the framework is bloated, or somebody wants the line count down. Neither survives contact with the week where retrieval quality drops and nobody can say whether it was the migration. If your only complaint is aesthetic, the money is better spent on evaluation.

One decision goes before everything else: are you removing the whole framework, or one of its four jobs? Keeping the orchestration and removing the integrations is a common, cheap and stable end state. “Remove the framework” is often the wrong scope for what is actually bothering you — the four jobs are set out in when a framework earns its place.

Inventory what it is actually doing

An hour with a grep and a spreadsheet, and it is the step that decides whether the rest goes well. You are looking for every place framework types cross into your code.

# Every import site, grouped — this is your work list.
rg -n '^\s*(from|import)\s+(langchain|llama_index|haystack)' --stats

# Where framework objects appear in signatures and annotations: the
# places a type leaked out of the seam and into your domain.
rg -n '(Document|BaseRetriever|Runnable|AgentExecutor|BaseMessage)' -t py

# Where prompts live. If this returns framework classes rather than
# your own files, prompts are the first thing to extract.
rg -n 'PromptTemplate|ChatPromptTemplate|from_messages'

Then classify every hit into exactly one of four buckets, and count them.

The counts tell you the shape of the job. Two hundred provider calls and one chain is a mechanical afternoon. Twenty provider calls and nine nested chains is a fortnight, and most of it is in one file.

Find the seams

A seam is a place where you can substitute an implementation without the caller noticing. If your codebase has none, the first commits of the migration are creating them — and those commits are safe, because they change no behaviour.

1. One function for model calls. Signature: messages in, text or a structured object out, plus usage. Initially its body calls the framework. Every call site moves to it. This commit is pure refactoring and can ship on its own.
2. One function for embeddings. Texts in, vectors out. Same pattern, smaller.
3. One function for retrieval. Query in, list of your own chunk type out — with text, source and score. Note your own type: the point of the seam is that the framework document class stops crossing it.
4. Prompts into files. Move every prompt string out of framework template objects into your own module or template files, and render them yourself. This is the highest-value early commit, because from here on you can read exactly what is being sent.

After these four commits nothing has been removed, no behaviour has changed, and the framework touches your application in four functions instead of forty files. That state is a legitimate stopping point: if the migration is deprioritised here, you have still fixed the thing that actually hurt.

The order of operations

Ordered by risk and reversibility, easiest and most reversible first, so that momentum builds and any step can be abandoned without stranding the work.

1. Replace the model call body. Swap the framework’s chat class for the provider SDK, or for a thin OpenAI-compatible client, inside your one function. Verify with a recorded set of inputs that outputs are equivalent — not identical, since sampling makes that impossible, but equivalent in shape, length and validation pass rate.
2. Replace embeddings. Critical caveat: use the same model and the same normalisation. A different embedding model means re-indexing the corpus, and doing that in the same change as the migration makes an inevitable quality question unanswerable.
3. Replace retrieval. Call your vector store’s client directly. Before merging, run your evaluation set through both paths and compare recall — this is the step most likely to change results silently, because framework retrievers apply defaults, score thresholds and filters you may not have known about.
4. Replace ingestion. Loaders and splitters. Only after retrieval, and only with a re-index you have planned, because changing chunk boundaries changes every vector.
5. Replace the control flow. Last, when everything underneath is yours. A chain becomes a function that calls your functions in order; an agent becomes the loop written out on the framework decision page.
6. Replace the cross-cutting concerns, then delete the dependency. Tracing, token counting and caching last, then remove the packages and let the lockfile diff tell you what else went with them.

Each of those is a mergeable change with a test. None of them requires a long-lived branch. That property is the entire method.

The translation table

The four hard parts

Streaming through composed steps

The genuinely non-trivial one. If your application streams tokens through a parser or through several composed steps, the framework was doing real work — incremental parsing, backpressure, propagating a cancellation from the client all the way to the provider. Budget for this specifically rather than discovering it. If you only stream plain text from one model call, it is an afternoon.

Retrieval defaults you did not know you had

Framework retrievers apply defaults: a value of k, a similarity threshold, sometimes a fetch-then-filter step or a diversity re-ranking. Reproducing the retriever without them changes results in a way that looks like the migration broke quality. Read the source of the specific retriever class you are replacing — it is usually under a hundred lines — and write the defaults down before you replace it.

Token counting and cost accounting

Whatever dashboards exist are reading the framework’s callback data. Provider responses carry usage, so the data is available, but the field names differ and the aggregation is yours to rebuild. Do it before the deletion and check the numbers overlap for a few days.

The one useful abstraction you will miss

Usually batching: running two hundred inputs with bounded concurrency, retries and ordered results. Write it once as a small utility. It is thirty lines with a semaphore and it is the one piece of framework machinery worth reimplementing rather than living without.

Knowing when you are done

Set the finish line before starting, or the migration acquires scope until it is abandoned. Four checks.

• The dependency is gone from the lockfile. Not unused — absent. An unused dependency still updates, still has vulnerabilities, and still constrains other versions.
• Evaluation numbers are within noise of the pre-migration run. You need the run from before. If you do not have one, take it now, before touching anything: this is the only artefact that answers “did we break it?” and it takes an afternoon. See building an eval harness.
• Traces and cost figures are continuous across the cutover. A gap in the dashboards is a migration that took the observability with it, which will be discovered during the next incident rather than now.
• The prompts are readable in the repository. If this is true and nothing else is, the migration was still worth it.

Not building a second framework

The predictable failure of this work is finishing it and discovering you have written a worse framework with no documentation and one maintainer. Four rules prevent it, and they are the whole of the discipline.

No abstraction with one implementation. A base class with a single subclass is speculation. Write the concrete thing; extract when there are two.

No registry, no plugin system, no dynamic dispatch by string. These are what made the framework hard to read. A dictionary of tool name to function is fine; a decorator that registers into a module-level table so that a config file can name things is the beginning of the same problem.

Functions over classes for anything stateless. Most of what a framework models as objects with lifecycle is a function that takes arguments and returns a value.

Keep the total under a thousand lines. That is roughly what the four jobs cost when written directly for one application. If it grows past that, either the application genuinely has framework-shaped complexity — in which case adopting one was right and a graph library may be the honest answer — or something is being generalised that has one user.