Building a Coding Agent That Can Run Tests

The loop is twenty lines and everybody writes it in an afternoon. What separates an agent that lands changes from one that burns forty dollars and deletes an assertion is four decisions around the loop, none of which are about the model.

The loop, and where it goes wrong

state = { failing: run_tests(), turns: 0, spend: 0 }

while true:
  edit   = model(task, context, state.failing)     # propose a patch
  applied = apply(edit)                            # <- format decides this
  if not applied.ok: continue_with(applied.error)  # a real turn, not a retry
  result = run_tests(scope_for(applied.files))     # <- sandbox and scope
  if should_stop(state, result): break             # <- the interesting part
  state = advance(state, result)

Each of the four commented lines is a decision with a wrong answer that looks fine in a demo and fails on a real repository. Taken in order.

Decision 1: the patch format

Whole-file rewrite

Simplest to apply and almost always wrong at scale. It costs the entire file in output tokens on every edit, it is bounded by the output token limit so it fails outright on large files, and it silently drops content the model did not think was important. A 900-line file that comes back as 840 lines has lost something, and nothing in the pipeline notices.

Unified diff

Cheap in tokens and the format everybody reaches for second. The problem is arithmetic: a hunk header like @@ -412,7 +412,9 @@ requires the model to count lines correctly in two coordinate systems, and models miscount. The result is the failure everyone who has tried this recognises:

patch: **** malformed patch at line 12: @@ -412,7 +412,9 @@
error: corrupt patch at line 34
error: patch failed: src/billing/invoice.ts:412
error: src/billing/invoice.ts: patch does not apply

It is survivable with fuzzy application — patch -l --fuzz=3 or a library that matches context lines and ignores the counts — but fuzzy matching introduces its own hazard, which is applying a hunk in the wrong place when the context appears twice.

Search/replace blocks — the one to use

Exact old text, exact new text, no line numbers to get wrong. Popularised by aider and now common; the important property is that it fails loudly and locally.

src/billing/invoice.ts
<<<<<<< SEARCH
    const total = items.reduce((a, i) => a + i.cents, 0);
    return total;
=======
    const total = items.reduce((a, i) => a + i.cents, 0);
    return applyRounding(total, rule.mode);
>>>>>>> REPLACE

Enforce two rules in your applier and the format becomes reliable. The SEARCH text must appear exactly once in the file — zero matches and one match are both fine outcomes to report, but two matches must be an error, because picking one is a coin flip that corrupts code. And a failed apply is fed back as a turn with the actual file content around the intended location, not retried blindly; the model usually fixes it immediately once it can see what is really there.

Decision 2: the sandbox

An agent that runs tests runs arbitrary code, and the code is determined partly by text it read from your repository. Treat it as untrusted execution, because it is.

• A container, with the repo mounted and nothing else. Not your laptop, not the CI runner that holds deploy credentials.
• No network by default. This one is not primarily about security. An agent with network access resolves a failing import by installing a package, which converts a bug into a new dependency and a green test. Allow the package registry explicitly if the task genuinely needs it; deny it otherwise.
• No credentials in the environment. A test suite that needs secrets should get fakes; a suite that talks to a real staging database in an agent loop will eventually truncate it.
• Wall-clock, memory and disk limits on every command. An infinite loop in generated code is not an unusual event.
• A spend ceiling checked every turn, not at the end.
• Work on a branch, commit every green. The commit is both the undo and the audit trail, and it makes the diff reviewable step by step instead of as one wall.

Sandboxing an agent covers the isolation choices in general; spend ceilings covers the case where the loop is the attack.

Decision 3: the test command and its output

Two things matter and both are about the loop’s cost. Scope the run: tests for the changed files first, full suite once before you stop. A forty-minute suite in the inner loop makes the agent useless regardless of how good the model is, and the build graph knows which tests could possibly be affected — see the monorepo queries on the large-codebase page.

Then truncate the output deliberately. A verbose pytest run against a large suite is tens of thousands of tokens of dots and collection noise, and feeding it back on every turn is the largest single line on the bill for a naive agent. Extract instead:

# what the model needs: the summary line, the names, and the first N failures
pytest -q --tb=short --maxfail=5 -p no:randomly 2>&1 \
  | tail -n 120 > /tmp/out.txt

# machine-readable is better than scraping, where the runner supports it
pytest --json-report --json-report-file=/tmp/r.json
jq '{summary, failures: [.tests[] | select(.outcome=="failed")
      | {nodeid, message: .call.longrepr}][0:5]}' /tmp/r.json

Give it the first few failures rather than all forty. Forty failures from one root cause is one piece of information repeated, and it crowds out the context the model needs to fix it.

Decision 4: stopping, properly

A turn limit is not a stopping condition, it is a cost cap. The conditions that matter are about progress and about integrity.

The no-progress condition deserves its own note because it is where almost all wasted spend goes. Compare failing-test sets across turns:

prev = state.failing            # a set of test node ids
now  = result.failing
if now == prev or (now - prev and not prev - now):
    state.stalled += 1          # nothing fixed, or fixed nothing and broke more
else:
    state.stalled = 0
if state.stalled >= 2:
    stop("no progress: " + ", ".join(sorted(now)[:3]))

And whatever stops it, the output is a branch and a diff for a human, not a merge. An agent that lands its own work removes the only step in the process that reliably catches the classes review is for.