Unit Testing Prompt Template Rendering With Every Variable Combination

Prompt rendering is the one part of an LLM application that is completely deterministic, needs no network, costs nothing to run, and is almost never tested. Every bug it can have reaches the model as a malformed prompt and comes back as a mysterious quality problem.

The renderer is a pure function, so test it like one

Separate rendering from calling. If your code interpolates variables inside the same function that builds the request and sends it, you cannot test rendering without a mock, and you will not write the test. Extract a function from template plus context to string, and the entire surface below becomes ordinary unit testing with no model in sight — the general argument in testing without the model.

Assert on the rendered string. Not on a hash of it, and not on a snapshot — snapshots of prompts are updated reflexively when they go red, which converts the test into a record of what happened rather than a statement of what should. Assert the specific properties that matter, listed below.

The values that break renderers

These are the inputs worth a named case each. Every one of them has a plausible route from a real user or a real database into your context object.

• Empty string. Renders to a prompt with a dangling label: Customer name: followed by nothing. The model will invent one. Decide whether empty is legal and assert the decision.
• Missing key entirely. Different from empty, and the difference is the whole of the unfilled placeholder bug. Assert it raises rather than renders.
• Whitespace only. Passes a truthiness check in most languages and is semantically empty.
• Very long value. A 200,000-character support thread in a field you sized for a sentence. Assert the renderer either truncates at a documented point or raises — silently producing a prompt that exceeds the context window is the worst option, and the one you get by default.
• A value containing the delimiter. If your template uses double braces, a value containing double braces will be re-scanned by some engines and left alone by others. You need to know which yours does.
• A value containing your section markers. If the prompt separates sections with a line like ### INSTRUCTIONS, a user who pastes that string is restructuring your prompt. This is prompt injection through the template layer — see prompt injection.
• Newlines and code fences. A value containing a triple-backtick fence will terminate a fenced block early and put the rest of your instructions inside a code span.
• Non-Latin script, combining marks, right-to-left text, and a lone surrogate. The last one is not exotic: it arrives from clients that slice strings by code unit.

Four properties that hold for every input

Named cases catch what you thought of. Properties catch the rest, and rendering is unusually well suited to property-based testing because the invariants are strong and the generator is cheap. Hypothesis provides the strategies: its strategy reference documents text(), fixed_dictionaries() and sampled_from(), which is all you need here.

from hypothesis import given, settings, strategies as st
from app.prompts import render, RenderError

context = st.fixed_dictionaries({
    "customer_name": st.text(),
    "order_id": st.text(),
    "history": st.text(max_size=5000),
})

@given(context)
@settings(max_examples=500)
def test_render_properties(ctx):
    try:
        out = render("support_reply", ctx)
    except RenderError:
        return                      # raising is a legal outcome; corrupting is not

    assert "{{" not in out          # no residual delimiter, whatever went in
    assert out.startswith(STATIC_PREAMBLE)
    assert out.count("### INSTRUCTIONS") == 1
    assert len(out) <= MAX_PROMPT_CHARS

• No residual delimiter. The rendered output never contains unrendered template syntax, for any context.
• Structure is preserved. The static frame of the prompt survives: the preamble is still first, each section marker appears exactly the number of times the template contains it. This is the property that catches injection through a variable, and it is stronger than any blocklist.
• Bounded output. Rendered length never exceeds a documented maximum. Either the renderer truncates or it raises; either is fine and the property is the same.
• Total or explicit. Every input either renders or raises a typed error. Returning None, an empty string, or a partially rendered prompt are all failures of this property, and all three are what an unconfigured engine does.

A value must never be a template

The severe bug in this area is a renderer that evaluates its own output. If a user-supplied value containing template syntax is re-rendered in a second pass, that user can read anything else in the context — other variables, and in some engines attributes of the objects they hold. This is a standard server-side template injection, and prompt templates are template engines.

Test it directly with a value that is a template referring to another variable, and assert the output contains the literal text rather than the other variable’s value. If it renders, the fix is structural: render in one pass, never feed rendered output back into the renderer, and if your pipeline genuinely needs two stages, use a distinct delimiter for the second so the first cannot produce syntax the second will act on.

Combinations without a combinatorial explosion

“Every variable combination” is not literally achievable — six variables with five interesting values each is 15,625 renders, and most of those pairs interact with nothing. Two techniques cover it properly.

Use pairwise coverage for the named edge cases: generate a set of contexts in which every pair of interesting values appears together at least once. That is a few dozen cases rather than thousands, and it catches essentially all real interaction bugs, because genuine three-way interactions between independent template variables are vanishingly rare. Then let the property test above handle the unbounded space, since it samples combinations you would not have enumerated.

One case deserves to be enumerated by hand regardless: the fully minimal context, where every optional variable is absent and every required one is at its shortest legal value. That is the prompt your model sees on a brand-new account with no history, it is the one your fixtures never resemble, and it is disproportionately where the rendered prompt turns out to be missing a section entirely.

Two things make this suite pay for itself beyond catching bugs. It runs in milliseconds with no network, so it can sit on every commit and give an answer before the developer has switched windows — which is the only reason anyone keeps a test suite. And it documents the contract of the template: reading the cases tells the next person which variables are required, which are optional, what happens to an oversized value and whether user content is trusted. That is information which otherwise lives in one person’s memory and gets rediscovered by whoever adds the ninth variable.