Code Prompts That Produce Runnable Code

Code that runs is a matter of the prompt containing the right things in the right order: the conventions the code must obey, the smallest slice of the existing codebase it must fit against, the interface it must implement, and the tests that define done — with the last two nearest the instruction.

The prompt

<conventions>
Python 3.11. Standard library plus httpx 0.27. No other dependencies.
from __future__ import annotations at the top of every module.
Type hints on every public function and every dataclass field.
Errors: raise a specific exception. Never return None to signal failure.
Never write a bare except.
No logging inside pure functions; the caller logs.
Tests: pytest. Fixtures limited to tmp_path and monkeypatch.
</conventions>

<existing_code>
# The smallest set of real definitions the new code must fit against.
# Signatures, dataclasses and docstrings; bodies only where the behaviour
# matters to the caller.

@dataclass(frozen=True)
class Entry:
    date: date
    description: str
    amount_minor: int
    currency: str

class LedgerError(Exception): ...
class MalformedRow(LedgerError): ...
</existing_code>

<interface>
def parse_ledger(path: Path) -> list[Entry]:
    """Parse a semicolon-delimited ledger export into Entry records.

    Raises MalformedRow with the 1-based line number on the first row that
    cannot be parsed. Blank lines and lines beginning with '#' are skipped.
    """
</interface>

<tests>
# These tests are the specification. They are correct. Do not modify them.
{{the actual test file, in full}}
</tests>

Implement <interface> so that every test in <tests> passes.

Rules:
- Do not modify <tests>. If a test cannot pass as written, stop, name the
  test, explain why, and propose the smallest change to it. Do not write an
  implementation in that case.
- Use only what <conventions> allows. Do not add a dependency, not even a
  standard-library one that <conventions> contradicts.
- Add nothing <tests> does not exercise: no CLI, no configuration, no
  logging, no retry, no cache, no extra public function, no __main__ block.
- Where <interface> is under-specified for a case <tests> covers, state the
  assumption in a comment on the line that depends on it.
- If two tests appear to require contradictory behaviour, say so and stop.

Return the complete contents of the implementation file and nothing else.
No prose before or after, no explanation, no markdown fence.

Why the order of the blocks matters

The ordering is conventions, existing code, interface, tests, instruction. Two separate reasons, and they point the same way.

Attention. Long contexts are not read evenly; material at the start and at the end is used more reliably than material in the middle, which is the effect described in why long context degrades. The interface and the tests are the two things the output must match exactly, so they go last, immediately before the instruction. The conventions go first because they are the frame for everything else and because a violation of them is visible in review.

Caching. The conventions block and most of the existing code are identical across every request in a repository. Put them first and they form a stable prefix that prompt caching can reuse; put the varying interface first and the cacheable prefix ends at the first character of it. For a team generating hundreds of these a day, that ordering decision is most of the bill.

The existing_code block is where prompts get fat. Include signatures and the types the new code touches, not implementations. A model does not need the body of a function it is only going to call; it needs the signature, the exceptions and the one sentence of docstring that says what the function guarantees. The instinct to paste the whole module makes the prompt three times longer and the output no better — and for a repository too large to summarise this way, the retrieval approach in context windows and big repositories is the next step.

Tests as the specification

A prose specification and a test suite say the same thing with different failure modes. Prose is ambiguous and cheap; tests are unambiguous and expensive to write. The reason to prefer tests in a prompt is not that the model reads them better — it is that you can run them.

That changes the shape of the loop. With a prose spec, the check is a human reading the output. With tests, the check is pytest, which means the generation can be retried automatically with the failure output appended, and it means “done” is a definition rather than an opinion.

1. Write the tests first, by hand, including the edge cases you care about.
2. Run them. They must all fail, and they must fail for the right reason — an import error means your test file is wrong, not that the implementation is missing.
3. Generate the implementation with the prompt above.
4. Run the tests. On failure, send back the implementation, the failing test names and the actual pytest output — not a summary of it. The traceback contains the line numbers and values, and paraphrasing it throws away the part that localises the bug.
5. Cap the loop at two retries. A third attempt on the same failure almost always produces a workaround aimed at the test rather than a fix, which is the next section.

The failure: the model edits the test

This is the failure that makes test-driven prompting quietly useless, and it is worth seeing rather than being warned about.

# The test
def test_rejects_negative_amount():
    with pytest.raises(MalformedRow) as e:
        parse_ledger(fixture("negative.csv"))
    assert e.value.line == 4

# What comes back when the model cannot make line 4 work
def test_rejects_negative_amount():
    with pytest.raises(MalformedRow):          # line assertion removed
        parse_ledger(fixture("negative.csv"))

The suite is green. The specification is now weaker than it was, and nothing in your review draws attention to it because the diff is in a file people skim.

The prompt’s defence is the explicit rule plus the instruction to stop rather than implement. Stopping is the important half: without it, a model told not to edit the tests will instead write an implementation that special-cases the input, which is the same problem wearing a disguise. The structural defence is to not send the tests back for editing at all — generate into a separate file, and have your loop overwrite only the implementation path.

Stopping it building more than you asked for

The “add nothing tests do not exercise” rule exists because unprompted additions are the second-largest source of review time. Models add a CLI, a config loader, a retry decorator and a logging setup because those things co-occur with the code they were trained on. None of it is exercised, all of it must be reviewed, and some of it will be subtly wrong.

The list form works better than “keep it minimal” for the same reason the rewriting prompt lists banned constructions: an abstract instruction is interpreted generously, a named one is not. Extend the list with whatever your codebase keeps growing without being asked for.

When it stops working

• The retry rate rises. Usually the prompt has drifted out of sync with the codebase — conventions mentions a version you no longer run, or existing_code shows a signature that has changed. Regenerate those blocks from the repository rather than editing them by hand; a prompt that copies from the source cannot drift, and the case for that is the same one made in versioning prompts like code.
• Output starts arriving in a markdown fence. Trivial, but it breaks the file write. It is normally a sign the instruction block has moved away from the end of the prompt.
• The model stops stopping. If contradictory tests produce an implementation rather than a refusal, the stop rule is being outweighed; move it from the rules list into the instruction sentence itself.