What Breaks in a RAG App's Prompt Template After a Provider Swap

The answers got worse after the swap: context ignored, citations dropped, facts appearing that are not in the retrieved chunks. Before rewriting the template, find out whether the model is even seeing the same chunks it saw last week.

Isolate retrieval from generation

A RAG pipeline has two model-shaped components and a provider swap frequently moves both. If the embedding model changed, the index is now being queried with vectors from a different space than the one it was built in, and the retrieved chunks are wrong before generation begins. No amount of prompt tuning fixes that, and every hour spent on the template is wasted.

Test it directly. Take fifty queries from your logs, and for each one record the chunk ids retrieved before the swap and after:

overlap = []
for q in queries:
    before = set(old_index.search(q, k=5))
    after  = set(new_index.search(q, k=5))
    overlap.append(len(before & after) / 5)

print("mean top-5 overlap:", sum(overlap) / len(overlap))

High overlap means retrieval is intact and the problem is in generation, which is what the rest of this page addresses. Low overlap means you have an index problem: either the embedding model changed, or it did not change but the query is being preprocessed differently, or the index was rebuilt with different chunking. Fix that first and re-measure before touching a word of the template.

The second isolation step is to freeze the chunks. Take a handful of queries where the answer got worse, capture the exact retrieved text, and replay it against both providers with the identical assembled prompt. If both give good answers from frozen chunks, the fault is upstream. If the new provider gives a worse answer from identical input, you now have a reproducible generation failure and a two-line test case for everything below.

Where the system prompt went

The most common generation-side cause is that the instructions moved, or partly vanished, in the adapter. Three variants, all of which produce the same symptom of the model apparently ignoring its rules:

• Dropped. An adapter that builds a message array for an API which takes the system prompt as a top-level parameter, and forgets to lift it, sends the instructions nowhere. The request succeeds. The model is answering with no system prompt at all. Log the outbound body once — this takes two minutes and resolves a surprising share of these investigations.
• Demoted. Lifted correctly into the parameter, but your rules about citations and refusals were written in a house style tuned for a model that weighted the system role differently. The text is present and less influential.
• Merged. Concatenated into the first user message because the target API has no separate slot in your adapter’s model. Now the instructions and the retrieved context are in the same block with no boundary between them, which is precisely the condition under which a model starts treating instructions as content to summarise.

Whichever it is, the correction is the same: put the instructions in the slot the target API defines for them, and re-read them as though you had never seen the old model’s behaviour. Instructions that worked by implication — because one model happened to prefer conservative answers — have to become explicit. The library’s general treatment is prompt portability.

Delimiters and where the model looks

A RAG prompt is a structured document pretending to be a string: instructions, k retrieved passages, metadata for citation, and the question. The model has to work out which is which from formatting alone, and models differ in what formatting they respond to. A template that used markdown headings and blank lines can, on a different model, lose the boundary between passage three and passage four — after which citations point at the wrong source, which is exactly the symptom teams describe as “it started making up citations”.

The robust form is explicit, machine-like structure with an identifier on every passage and an instruction that names the identifier:

<context>
<doc id="7" source="refunds-policy-v3.md">
Refunds are issued to the original payment method within 14 days.
</doc>
<doc id="12" source="shipping-faq.md">
Orders ship within two business days.
</doc>
</context>

Answer only from the documents above. After each claim, cite the document
that supports it as [doc:ID]. If the documents do not contain the answer,
reply exactly: NOT_IN_CONTEXT

Three properties make this survive a provider change. The boundaries are unambiguous tokens rather than whitespace conventions. The citation format is a fixed string the model copies rather than a style it has to infer, and it is machine-checkable, so a regression test can assert every citation refers to a document that was actually supplied. And the abstention case has a literal output, which turns “the model hallucinated” into a testable condition. See the RAG citations page for the wider argument, and RAG generation failures for the failure taxonomy this sits inside.

Position matters as well, and it is cheap to test. Instructions after the context rather than before it is a one-line change and frequently moves the result, because the last thing in a long prompt is reliably attended to. If your template puts a page of retrieved text between the instruction and the question, try restating the instruction at the end before concluding the model cannot follow it.

The three defaults that changed underneath you

Temperature. Providers do not share a default, and the ranges differ — one API accepts up to 2 while another caps at 1, so a value carried across can be rejected outright with an error saying temperature must be between 0 and 1, or accepted and mean something different. A RAG system usually wants this low and explicit. Set it.

The output limit. The parameter is variously max_tokens, max_completion_tokens and max_output_tokens, and on at least one API it is required rather than optional. Omit it where it defaults low and answers are truncated mid-sentence; carry over a small value and the same thing happens. Check the finish or stop reason on a failing request: a value meaning length tells you this is the whole bug.

Stop sequences. If your template relied on a stop string to end the answer, confirm the parameter name and the limit on how many are accepted survived the move. A stop sequence that is silently not applied produces answers that run on into invented follow-up questions, which reads like a quality regression and is a configuration one.

Add to that the case where output is parsed: if the template asked for JSON and you were relying on one provider’s schema enforcement, the target’s structured-output mode has different strictness rules, and a template that never needed a repair step now needs one.

What to adjust, in order

1. Confirm retrieval is unchanged, by chunk-id overlap. If it is not, stop and fix the index.
2. Log one full outbound request and read it. Confirm the system prompt is present, in the right slot, and that the chunks are where you think.
3. Set temperature, the output limit and stop sequences explicitly rather than inheriting defaults.
4. Make the structure explicit: delimited passages with ids, a literal citation format, a literal abstention string.
5. Move the instruction to after the context and re-measure.
6. Only now rewrite the wording, one change at a time, against your golden set — not against a handful of queries you tried by hand. The library covers building that set in the golden dataset page.

Each step is a measurement rather than an opinion, and running them in this order means you stop as soon as the numbers recover instead of rewriting a template that was never the problem.