Turning a Working Prompt Into a Reusable Recipe

A prompt in a string constant is not maintainable by anyone but its author, and often not by them six months later. What makes it maintainable is a short, fixed set of metadata — what it depends on, how it is evaluated, what it costs, and how you will know it has broken.

The format

One YAML file per prompt, next to the prompt text, in the same repository as the code that calls it.

id: support-triage
version: 4
owner: support-platform
updated: 2026-08-04

task: >
  Classify an inbound support ticket into one of six types, set an urgency,
  and return the span that establishes each.

prompt: prompts/support-triage.v4.txt

inputs:
  ticket_text:
    type: string
    max_tokens: 4000
    source: zendesk.ticket.description
    truncation: head          # what to drop when it does not fit, and why
  teams:
    type: enum_list
    source: config/teams.yaml

output_schema: schemas/triage.v3.json

model:
  intended: any instruction-following model with reliable JSON output
  tested_on:
    - <the models you actually ran the eval set against, with dates>
  parameters:
    temperature: 0
    max_tokens: 400
  known_bad:
    - <any model or setting where this failed, and how it failed>

depends_on:
  - labels.md - the six label definitions. Changing a definition invalidates
    the eval set below.
  - schemas/triage.v3.json - adding a required field is a breaking change for
    every consumer.
  - routing.py - consumes type and urgency. New enum values need a branch
    there first.

eval:
  set: evals/triage-120.jsonl
  metric: exact match on type; urgency within one level
  gate: type accuracy >= 0.88 AND no regression on the 12 pinned cases
  pinned: evals/triage-pinned-12.jsonl

fails_when:
  - the other rate over a week rises above 8% - a new ticket category
    exists that the labels do not cover
  - evidence spans stop being substrings of the ticket - usually a model
    change; check before editing the prompt
  - tie_break_used rises above 15% - the definitions have stopped doing the
    work and the tie-breaks are carrying it

cost:
  tokens_in: ~1,400 fixed + the ticket
  tokens_out: ~90
  notes: fixed portion is cacheable; keep the ticket last in the prompt

change_log:
  - v4 (2026-08-04): added the billing/bug tie-break. Other rate fell on the
    eval set.
  - v3 (2026-05-11): added the evidence span. Output tokens up ~12.
  - v2 (2026-03-02): split account_access out of how_to.

Twelve keys. It takes twenty minutes to write for a prompt you already have, and the sections below are the four that repay it.

The field nobody writes

fails_when is the difference between documentation and an operational artefact. Every other field describes what the prompt is; this one tells the person on call what to look at.

The author of a prompt knows things nobody else can reconstruct: which output field goes strange first, which metric moves before users complain, which failure looks like a prompt problem and is actually a model change. That knowledge decays within weeks and is the single most expensive thing to rediscover.

Write each entry as a signal, a threshold and a first action.

• Signal. Something already logged. If it is not logged, log it, or the entry is a wish.
• Threshold. A number. “Rises” is not actionable at 3am; “above 8% over a week” is.
• First action. What to check first, which is usually not the prompt. “Check whether the model version changed” has saved more hours than any prompt edit.

Every drift section in this cluster is a fails_when list. Extracting them into the recipe file is what makes them survive the person who wrote them.

Dependencies are invisible without this

A prompt is coupled to things outside itself, and none of the couplings show up in a call graph:

• A label or field definition file. Editing one definition changes the meaning of the output and invalidates every example in the eval set. Nothing warns you.
• An output schema. Consumers parse it. A field that becomes optional breaks a consumer that assumed it.
• A downstream branch. A new enum value from the model hits a routing function with no case for it — and the failure appears in the router, several services from the prompt that caused it.
• A retrieval index or a vocabulary list. Reindexing changes what the prompt sees, and the prompt is unchanged and the behaviour is not.

Write the coupling and the consequence, not just the filename. “labels.md — changing a definition invalidates the eval set” is a sentence that stops somebody, which is the point. Add a reverse pointer in the depended-on file too: a comment at the top of labels.md saying which prompts read it.

Tested on, not works with

tested_on records what you ran, with dates. It is not a compatibility claim, and the distinction matters because a compatibility claim ages into a false statement while a record of what was run stays true forever.

known_bad is the more valuable half and it is nearly always empty because nobody thinks to write down a negative result. “This model returns the JSON wrapped in a markdown fence” is thirty seconds to record and an afternoon to rediscover. Write it with the symptom, not just the verdict, so the next person can tell whether they are seeing the same thing.

Keep intended as a capability statement rather than a model name: “any instruction-following model with reliable JSON output” stays true as models are replaced. Model names go in tested_on, where they are dated facts rather than assertions.

Keeping the metadata honest

Documentation that is not checked drifts from the thing it documents. One check catches most of it: every placeholder in the prompt text must be declared in inputs, and every declared input must be used.

import re, sys, pathlib
import yaml   # pip install pyyaml

PLACEHOLDER = re.compile(r"\{\{\s*([a-zA-Z0-9_]+)\s*\}\}")

def check(recipe_path: str) -> dict:
    meta = yaml.safe_load(pathlib.Path(recipe_path).read_text())
    text = pathlib.Path(meta["prompt"]).read_text()
    used = set(PLACEHOLDER.findall(text))
    declared = set(meta.get("inputs") or {})
    missing_files = [p for p in [meta.get("output_schema"),
                                 (meta.get("eval") or {}).get("set")]
                     if p and not pathlib.Path(p).exists()]
    return {"undeclared_placeholders": sorted(used - declared),
            "unused_inputs":           sorted(declared - used),
            "missing_files":           missing_files,
            "version_in_prompt_path":  str(meta["version"]) in meta["prompt"]}

if __name__ == "__main__":
    bad = False
    for path in sys.argv[1:]:
        r = check(path)
        if r["undeclared_placeholders"] or r["unused_inputs"] or r["missing_files"]:
            print(path, r); bad = True
    sys.exit(1 if bad else 0)

Run it in CI over every recipe file. It catches the two changes that break a prompt silently — a placeholder added to the text and never wired up, and an input removed from the caller while the text still references it — and it costs nothing.

The eval gate is the other half of the check, and it belongs in the same CI job: no prompt change merges without the eval set passing its threshold and the pinned cases holding. The pinned set is the important one — a dozen cases that must never regress, chosen because each represents a bug you have already fixed once. The general shape of that workflow is regression testing for prompts and models.

Where it lives

1. In the repository, next to the code that calls it. A prompt in a SaaS console is a production change with no diff, no review and no way to bisect. Keep the text in a file so a change shows up in a pull request like any other.
2. Version in the filename. support-triage.v4.txt rather than a file that changes under a fixed name. Old versions stay readable, and a log line saying v3 can still be matched to text.
3. The version in every log line. Without it you cannot answer “did this behaviour change when we changed the prompt”, which is the first question of every investigation.
4. The eval set in the same repository. A gate whose test data lives somewhere else is a gate that gets skipped.
5. One owner, named. Not a team alias that resolves to nobody. The owner is who decides whether a proposed change is worth re-running the eval set for.

The argument for treating prompts as source rather than configuration is made at length in versioning prompts like code; this format is one concrete way to do it. Adopt the four fields that pay for themselves — fails_when, depends_on, eval.gate and change_log — and add the rest when you find you need them.