Testing That a Prompt Template Doesn't Leak an Unfilled Placeholder

A customer email arrives beginning “Dear {{customer_name}}”. The template rendered, the request succeeded, the response was well formed, and nothing in your logs is red. The bug is that a key was missing from the context and your renderer treated that as acceptable.

The symptom, and the worse version of it

The visible form is a literal placeholder in output that reached a user. Embarrassing, easy to diagnose from the string, and usually a five-minute fix once found.

The version that matters more is the one where the model helps. Given a prompt containing Customer name: {{customer_name}}, a capable model will frequently infer that this is a template artefact and produce a plausible name, or address the customer generically, or silently drop the salutation. The output looks correct. No string search will find it. What you have shipped is an email addressed to a person whose name the model made up, and the only trace is in the prompt, which you may not be logging.

That asymmetry decides where the assertion goes. Asserting on output catches the visible half and misses the dangerous half. Assert on the rendered prompt, before it is sent.

Why the renderer let it through

Nearly every template engine defaults to permissive: a missing key renders as empty, or renders as the literal placeholder, and neither raises. That default is correct for a web page where a missing subtitle should not take down the site, and wrong for a prompt where a missing variable changes the instruction the model receives.

The missing key usually arrives from one of four places, and it is worth knowing which:

• A new variable was added to the template and one of several call sites was not updated. Most common by a distance, and invisible in review because the diff touches only the template file.
• A conditional branch builds a smaller context — the anonymous user path, the retry path, the batch path.
• A rename where the template says customer_name and the context now says customerName.
• A multi-stage chain where stage two renders a template whose context depends on a field stage one did not produce. This is the hardest to find because everything upstream succeeded.

Make the renderer strict

The first fix is to stop the renderer from tolerating the condition. In Jinja2 this is a one-line configuration: constructing the environment with undefined=StrictUndefined makes any reference to an undefined name raise UndefinedError at render time rather than producing an empty string.

from jinja2 import Environment, FileSystemLoader, StrictUndefined
from jinja2.exceptions import UndefinedError
import pytest

env = Environment(loader=FileSystemLoader("prompts"), undefined=StrictUndefined)

def test_missing_variable_raises_rather_than_rendering():
    template = env.get_template("support_reply.j2")
    with pytest.raises(UndefinedError):
        template.render(order_id="4471")     # customer_name deliberately absent

In a language without a strict mode available, the renderer becomes your own function: collect the keys the template references, diff them against the keys the context supplies, and raise on any difference. Do the diff in both directions. A context key with no corresponding placeholder is also a bug — it means a template edit dropped a variable somebody is still computing, and that variable’s content is now silently absent from the prompt.

The boundary guard, and testing the guard

Strict rendering handles variables the engine knows about. It does not handle a placeholder that arrived as literal text — a template fragment pulled from a database, a partial concatenated by hand, a prompt assembled in two systems. So add a second, dumber check at the last point before the request leaves: scan the final prompt string for residual template syntax and refuse to send.

const RESIDUAL = /\{\{\s*[\w.]+\s*\}\}|\$\{\s*[\w.]+\s*\}/;

export function assertNoResidualPlaceholder(prompt: string): void {
  const hit = RESIDUAL.exec(prompt);
  if (hit) {
    throw new UnrenderedPlaceholderError(
      `prompt still contains ${hit[0]} at index ${hit.index}`,
    );
  }
}

Then test the guard, not only the renderer. This is the part that gets skipped, and a guard nobody tested is a guard that has a typo in its regular expression. Give it the forms it must catch — double braces, double braces with inner whitespace, dotted paths, dollar-brace syntax if your codebase uses it — and, just as importantly, the forms it must not catch, so that a future tightening does not start rejecting legitimate prompts.

it.each([
  "Dear {{customer_name}},",
  "Dear {{ customer_name }},",
  "Order {{order.id}} is ready",
  "Hello ${user.name}",
])("rejects %j", (prompt) => {
  expect(() => assertNoResidualPlaceholder(prompt)).toThrow(UnrenderedPlaceholderError);
});

it.each([
  "Return JSON like {\"name\": \"Ada\"}",
  "Use the set notation {1, 2, 3} in your answer",
  "The customer wrote: use {{ }} for templating",
])("allows %j", (prompt) => {
  expect(() => assertNoResidualPlaceholder(prompt)).not.toThrow();
});

When the braces came from the user

The negative cases above are the reason this guard needs care. A customer support prompt legitimately contains user text, and users write braces: JSON in a bug report, set notation, a snippet of somebody else’s template. A guard that scans the entire prompt will eventually reject a real request, and the fix applied under pressure will be to delete the guard.

Two ways out, and they are not equivalent. The narrower guard runs the check on the template’s static regions only — render with a sentinel context first, note where the interpolations land, and scan everything outside those spans. Correct, and more machinery than most teams want.

The simpler and usually better route is to make your placeholder syntax unmistakable. If templates use a delimiter no human types — a doubled sentinel with a marker character, rather than double braces — then any occurrence in a final prompt is unambiguously yours, the guard has no false positives, and the negative test cases above stop being a design problem. This costs one migration and removes the entire class.

Whichever you choose, keep the guard’s failure loud and specific. It should name the placeholder, the template, and the call site, so the person reading the alert knows which of the four causes in the section above they have without reproducing anything. And log the rendered prompt on failure — redacted, following whatever your policy is for prompt content in logs, but logged, because a guard that fires and discards its evidence has only converted a silent bug into a mysterious one.

One last decision to make explicitly rather than by default: what the guard does in production when it fires. Raising means the user gets an error instead of a wrong answer, which is usually right for anything transactional and clearly wrong for a background summarisation job that will simply retry into the same failure forever. The middle option — fall back to a version of the prompt with the optional section removed — is defensible where the placeholder is in a section that can be dropped, and indefensible where it is in the instruction. Whichever you choose, write a test for the production path too. A guard whose only tested behaviour is throwing in a unit test will do something unrehearsed the first time it fires against real traffic.