Meta-Prompting: Using an LLM to Write Your Prompts

Prompt optimisation by machine has a real literature and a real failure mode, and they are the same thing viewed from two distances: a search process that will happily find whatever your dev set rewards, including noise.

The idea, and why it is not silly

A prompt is a string scored by a metric. Anything that proposes strings and keeps the ones that score well is an optimiser, and a language model is an unusually good proposer because it generates plausible variations rather than random ones.

The term covers two activities that are worth separating. One is asking a model to draft a prompt from a description of the task — useful, entirely unmeasured, and about as reliable as asking it for any other first draft. The other is the optimisation loop below, which has a metric and a search, and therefore has failure modes that can be reasoned about rather than merely regretted. Enthusiasm for the first is routinely quoted as evidence for the second.

Zhou et al. (2022), Large Language Models Are Human-Level Prompt Engineers, formalised this as APE: generate candidate instructions from a few input-output demonstrations, score each on a held-out set, keep the best. They reported machine-found instructions matching or beating human-written ones across a range of tasks. Yang et al. (2023), Large Language Models as Optimizers, went further with OPRO, feeding the optimiser its own history of instructions and their scores and asking for a better one.

The OPRO result is the one everybody quotes and the one worth understanding properly. For one model on one benchmark, the best instruction it discovered was Take a deep breath and work on this problem step by step, and the paper reports it scoring around 80% on GSM8K against about 72% for the familiar Let’s think step by step — a real gap, for a phrase no human would have proposed for a reason.

The loop

train, dev, test = split(cases, 0.5, 0.25, 0.25)   # test opened once, at the end

best = (baseline_prompt, score(baseline_prompt, dev))
history = [best]

for round in range(N_ROUNDS):
    failures = sample(errors_of(best.prompt, train), k=8)
    candidates = optimiser_model(
        instruction="Rewrite the prompt so these cases succeed. "
                    "Keep the output contract identical.",
        prompt=best.prompt,
        failures=failures,          # the actual inputs and wrong outputs
        history=history,            # prompt -> score, so it can see the trend
        n=6,
    )
    scored = [(c, score(c, dev)) for c in candidates]
    history += scored
    best = max(history, key=lambda h: h[1])

report(score(best.prompt, test))   # the only number you are allowed to believe

Two details do the work. Feeding actual failing cases rather than a description of the problem is what makes the rewrites specific — this is the core of Pryzant et al. (2023), which treats a summary of errors as a textual gradient and does beam search on it. And keeping the history of scores gives the optimiser a direction rather than a single point.

Where it overfits

Take the OPRO result at face value and its implication is uncomfortable: the winning instruction is semantically arbitrary. It encodes nothing about the task. That is the signature of a search that has found a quirk of one model’s conditioning rather than a fact about the problem — and quirks do not transfer. An optimised prompt is tuned to a model version, a decoding configuration, and the distribution of your dev set, and any of the three changing can return it to baseline.

Two symptoms tell you it has happened. The optimised prompt contains instructions that are oddly specific to cases in your dev set (“if the invoice is from a German entity, check the VAT line first” when only three dev cases are German). And its advantage disappears on held-out data — which you only find out if you kept some.

Budget the search before starting it, too. Six candidates per round, ten rounds, a 200-case dev set and one call per case is 12,000 evaluation calls plus the optimiser’s own; at a cent each that is a bill somebody should approve deliberately. Score on a stratified subsample during the search and only score the finalists on the full set — the search does not need precision, the decision does.

The arithmetic of a fake win

This is the part the tooling never mentions. Suppose your dev set has 100 cases and true accuracy is around 80%. The standard error of a proportion is √(p(1−p)/n) = √(0.8·0.2/100) = 0.04 — four points. A candidate that scores four points better than the baseline is one standard error away, which is to say indistinguishable from a coin landing your way.

Now generate six candidates per round for ten rounds. You have run sixty comparisons, so the best of them is expected to be well above the true best by chance alone; this is the multiple-comparisons problem with the optimiser as an enthusiastic accomplice. The discipline that survives it:

• Three splits, not two. Optimise against train, select on dev, report on a test set opened once. If you look at test twice, it is a dev set.
• Size the dev set for the effect you care about. Detecting a genuine three-point gain at 80% baseline takes thousands of cases per arm, not a hundred. If you cannot afford that, only adopt large wins.
• Use paired comparison. Score both prompts on the same cases and count only the cases where they differ. Pairing removes case difficulty from the variance and needs far fewer examples than two independent proportions.
• Re-run the whole thing on a model change. An optimised prompt is a fitted parameter and the model is part of the fit.

What to actually use it for

Used as an oracle it disappoints. Used as a generator of candidates for a human to judge, it is genuinely good at three things: proposing rewrites you would not have thought of, converting a pile of failure cases into specific instruction edits, and writing the tedious parts — enumerating edge cases, drafting an output contract, reformatting a prompt into a family’s conventions.

The honest framing is that meta-prompting compresses the search, not the judgement. Every candidate it produces still has to survive an eval you would have trusted before you saw the result.

Decide up front who owns the output. An optimised prompt nobody can explain becomes a maintenance problem the first time it needs an exception added, which is why the most durable use of these loops is to produce candidate edits that a person folds into a prompt they still understand.