DSPy: Compiling Prompts Instead of Writing Them

DSPy’s claim is that hand-written prompts are a local optimum reached by a human guessing, and that if you can score an output automatically you can search for a better prompt instead. That is a real idea and it works. It also means the page’s first job is the metric, because without one there is nothing to search against.

The idea, in one paragraph

You declare what a step takes and returns — the signature. You choose a strategy for satisfying it — the module. You supply examples and a metric. An optimiser then searches over prompts and demonstrations, scoring candidates with your metric, and produces a version of your program with the winning instructions and examples baked in. You never write the prompt string. You write the contract and the scorer.

Write the metric first

This is the whole discipline of the library and it is the step people skip. An optimiser is a hill climber; the metric is the hill. If the metric is loose, you will get a program that scores well and is worse in production, and you will not find out for weeks.

def metric(example, pred, trace=None) -> float:
    # 1. Hard constraints first: a wrong shape is a zero, not a partial score.
    if not pred.answer or len(pred.answer) > 400:
        return 0.0

    # 2. The thing you actually care about.
    correct = normalise(pred.answer) == normalise(example.answer)

    # 3. A second term, if quality has more than one dimension.
    grounded = all(c in example.context for c in pred.citations)

    return 1.0 if (correct and grounded) else 0.0

Two properties make a metric usable. It must be cheap, because it runs hundreds or thousands of times per optimiser run — a metric that calls a large model for every judgement can cost more than the optimisation itself. And it must be hard to satisfy accidentally: if a program can score 0.8 by returning a fixed string, the search will find that before it finds anything useful. The failure mode is exactly specification gaming in miniature, and it is common enough that you should check what the top-scoring candidate actually outputs before believing the score.

Signatures and modules

The declarative core has been stable since the library was named DSPy. A signature can be written as a string for simple cases or as a class when the field descriptions matter — and the descriptions do matter, because they are part of what the optimiser is optimising.

class ExtractVerdict(dspy.Signature):
    """Decide whether the context supports the claim."""
    context: str = dspy.InputField(desc="Source passages, newline separated")
    claim:   str = dspy.InputField()
    verdict: str = dspy.OutputField(desc="one of: supported, refuted, unclear")
    quote:   str = dspy.OutputField(desc="the sentence that decided it")

class FactChecker(dspy.Module):
    def __init__(self):
        super().__init__()
        self.check = dspy.ChainOfThought(ExtractVerdict)

    def forward(self, context, claim):
        return self.check(context=context, claim=claim)

A program is ordinary Python. Branching, loops, calls to your database and calls to other modules all live in forward, and the optimiser sees the module calls inside it. That is the property that makes multi-step programs optimisable rather than only single prompts.

What compiling actually does

Compilation is not code generation. It is a search, and it is worth knowing the shape of it because that is what you are paying for.

1. Bootstrap demonstrations. Run the current program over training examples, keep the traces where the metric scored well, and use those as few-shot examples for the modules inside. Self-generated demonstrations, filtered by your metric.
2. Propose instructions. The stronger optimisers also generate candidate instruction texts for each signature, using a model, informed by the task and the data.
3. Evaluate candidates. Each combination of instructions and demonstrations is scored on a validation split. This is where the model calls are spent.
4. Keep the best and return a new program object. Same class, same forward, different baked-in prompts.

The compiled artefact is a build output

The output of compilation is a set of instructions and demonstrations that can be saved and loaded. Treat it as a build artefact, not as a notebook side effect, or you will end up unable to reproduce the program running in production.

• Save it to a file and commit it, or store it somewhere versioned. It is the thing that actually determines behaviour.
• Record what produced it: the training data snapshot, the metric version, the optimiser and its budget, the model used for optimisation and the model used for inference. Those five facts are the difference between a reproducible artefact and a lucky run.
• Re-compile when the model changes. A program optimised against one model carries demonstrations tuned to its quirks. Swapping the inference model without re-compiling is reasonable but not free, and it is worth re-running the evaluation rather than assuming.
• Keep the evaluation split out of the optimiser. Train, validate, test — the ordinary discipline. An optimiser that has seen your test set has told you nothing.

What an optimiser run costs

Nobody publishes a number for your task, and any number quoted for somebody else’s is not transferable. The arithmetic, though, is straightforward and you should do it before starting a run rather than after reading the bill.

calls ≈ candidates × validation_examples × modules_per_program
      + bootstrap_examples × modules_per_program
      + instruction_proposals            (a handful, on a stronger model)

worked example, all values chosen by you:
  candidates            = 12
  validation examples   = 100
  modules per program   =   2
  bootstrap examples    =  50

  12 × 100 × 2 = 2,400 evaluation calls
   50 × 2      =   100 bootstrap calls
                ≈ 2,500 calls per optimiser run, plus metric cost

at 900 input + 250 output tokens per call:
  2,500 × 1,150 ≈ 2.9M tokens per run

Two conclusions fall out. Optimise with a small fast model in the inner loop wherever the metric allows it, because that multiplier is applied to every candidate. And if your metric is itself a model call, you have just doubled the count — which is why cheap deterministic metrics are worth a great deal more than they look. Evaluation set size matters here too: a hundred validation examples is enough to rank candidates that differ substantially and not enough to distinguish two that differ by a point.

When DSPy is the wrong tool

If you cannot write an automatic metric, DSPy has nothing to offer. Tasks judged on tone, taste or open-ended helpfulness fall here, and a model-graded metric for them is usually just a second opinion with the same biases as the first.

If you have fewer than about fifty labelled examples, the search has too little to go on and will overfit the handful you have. Collect data first; the collection is the valuable part regardless of what you do with it afterwards.

And if the prompt needs to be read and edited by non-engineers — support, legal, a domain expert — a compiled artefact full of machine-selected demonstrations is the wrong medium. In that case keep a hand-written prompt under version control and use evaluation to check changes rather than to generate them.