Migrating a LangChain Agent to a Different Tool-Calling Provider

The agent still runs. It just calls the wrong tool, or calls the right tool with the wrong arguments, or answers from memory when it should have looked something up. Three distinct causes produce that, and only one of them is about the model.

The constructor was provider-specific

The first thing to check costs nothing. LangChain has carried several agent constructors whose names encode a provider — create_openai_functions_agent and create_openai_tools_agent among them — and a provider-neutral create_tool_calling_agent that dispatches through the chat model’s own bind_tools implementation. If your agent was built with one of the OpenAI-named constructors, it is emitting a request shaped for the OpenAI function-calling surface regardless of which model object you handed it.

What happens next depends on the new integration. If it does not implement the interface at all you get a clean NotImplementedError and you know immediately. The worse case is that it does implement it, the request goes out, and you get plausible behaviour that is subtly worse — which is the situation everyone actually reports. Swap the constructor to create_tool_calling_agent first, then judge the model.

The same applies one layer down. If your code calls bind_tools with a tool_choice argument, check what value you are passing. The vocabularies differ between providers: the OpenAI surface documents auto, none, required and an object naming a specific function, while Anthropic’s tool-choice object uses types including auto, any and tool. LangChain normalises some of these and passes others through. A string that meant “you must call some tool” on one provider is not guaranteed to mean anything on the next, and a tool-choice constraint that silently stops applying is exactly the kind of change that shows up as degraded selection.

The prompt needs a scratchpad placeholder

The second failure announces itself:

ValueError: Prompt missing required variables: {'agent_scratchpad'}

— sometimes naming tools and tool_names as well, depending on which constructor you moved to. This is validation running before the agent is built, comparing the constructor’s required variables against your prompt’s input_variables. It is not a model problem and no amount of prompt tuning will fix it.

The distinction that matters is between the two families. A ReAct-style constructor wants tools and tool_names interpolated into a string prompt and a string scratchpad, because that agent describes its tools in prose and parses the model’s text output. A tool-calling constructor wants the tools bound to the model as a structured schema and the scratchpad as a MessagesPlaceholder, because the intermediate steps are real assistant and tool messages rather than formatted text.

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a support agent. Use the tools rather than guessing."),
    MessagesPlaceholder("chat_history", optional=True),
    ("human", "{input}"),
    MessagesPlaceholder("agent_scratchpad"),   # messages, not a string
])

Mixing the two — a ReAct prompt with a tool-calling agent — produces an agent that is told about its tools twice, once in prose and once as a schema, and that is a genuine cause of worse selection. The model sees two descriptions that disagree in wording, and the prose one is usually the stale one.

Replayed history carries the old shape

The third cause is the one that is hardest to see, because it only affects conversations that existed before the migration. Agent transcripts contain assistant turns with structured tool calls in them, and those turns were serialised in the previous provider’s shape.

Replaying them into a new provider goes wrong in two ways. The identifiers do not match the new provider’s expectations — the OpenAI shape carries a tool_calls array on the assistant message with an id per call, answered by messages with role tool carrying tool_call_id; Anthropic carries a tool_use content block with an id, answered by a tool_result block in the following user turn. And the pairing rules are stricter on one side: a tool_use block that is not answered by a matching tool_result in the next turn is an error, where a dangling tool call in the OpenAI shape often survives. Any transcript truncated mid-loop — by a context-window trim, by a crash, by a user cancelling — contains exactly that dangling pair.

The fix is to normalise transcripts at the storage boundary rather than at the send boundary: store your own neutral representation of a step (tool name, arguments, result, error) and render it into the current provider’s shape at call time. That also makes the trim logic honest, because you can drop a whole step rather than half of one. The field-by-field mapping of a tool-call sequence is its own page; the agent-specific point is that your history store is the thing that has to change, not your agent.

Tool selection genuinely differs

Once the three mechanical causes are eliminated, there is a real residue. Models differ in how eagerly they call tools, how they behave when two tool descriptions overlap, whether they emit several tool calls in one assistant turn, and how strictly they adhere to a declared argument schema. These are consequences of how each model was post-trained for tool use, and no amount of adapter code removes them.

The concrete things to look at, in the order they are worth checking:

• Parallel calls. Whether the model emits one tool call per turn or several changes the shape of your loop and the number of round trips. Where a provider supports disabling it, the OpenAI surface exposes a boolean for parallel tool calls; where it does not, your loop has to tolerate both. An agent written assuming one call per turn will drop the second one silently.
• Schema strictness. Whether the provider enforces your JSON Schema or merely shows it to the model decides whether you need argument validation before execution. You need it anyway, but the failure rate you are validating against changes.
• Description sensitivity. Tool descriptions written against one model tend to be tuned to its habits — often a single sentence that was enough because that model already inferred the rest. A new model gets only what is written.
• The empty-answer case. Some models answer directly where others call a tool first. If your product depends on a citation or a lookup happening, an eager direct answer is a correctness bug even when the answer is right.

Measuring it on your own tools

Nobody else’s tool-selection benchmark answers your question, because the thing that varies is your tool descriptions. Build the smallest evaluation that does:

1. Take fifty real user turns from your logs, spread across the tools you actually have, including at least ten where the correct behaviour is to call no tool at all.
2. Label each with the tool that should be called and the arguments that should be passed. Argument correctness is a separate score from selection correctness; keep them separate.
3. Run the agent for one step only — bind the tools, send the turn, capture the tool call, execute nothing. A single-step harness is enough to score selection and removes the noise of a full loop.
4. Score both providers on the same set. Report the confusion between tools rather than one accuracy number: which tool is being chosen instead of which tells you what to rewrite.
5. Rewrite the descriptions of the confused pair, and re-run. This is usually where the difference goes, and it is cheaper than changing provider back.

Keep the harness after the migration. It is the same harness that catches a selection regression when the provider updates the model underneath you, which is the identical failure arriving without a deploy to blame it on.