SWE-bench and Friends: Do Coding Benchmarks Predict Anything?

No score appears on this page. That is not coyness — by the time you read a number it belongs to a model version, a scaffold and a leaderboard revision that have all moved. What is durable is how the benchmark is built, and that tells you exactly what it can and cannot predict.

What SWE-bench actually is

Jimenez, Yang, Wettig, Yao, Pei, Press and Narasimhan introduced it in “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (ICLR 2024). The construction is the interesting part:

• 2,294 instances mined from the history of 12 popular Python repositories — Django, scikit-learn, sympy, matplotlib and similar.
• Each instance is a merged pull request that closed an issue and added or changed tests. The repository at the PR’s parent commit is the starting state; the issue text is the prompt.
• The task is to produce a patch. It is graded by running tests, not by comparing to the human patch: FAIL_TO_PASS is the set of tests the real PR made pass, and they must now pass; PASS_TO_PASS is everything that already passed, and it must still pass.

That grading design is genuinely good, and the PASS_TO_PASS half is the bit worth stealing — it is a regression check built into the score, and it is the reason a solution that deletes the failing assertion does not count. Copy it into your own agent’s stopping condition.

The most instructive thing about SWE-bench is the existence of SWE-bench Verified: a 500-instance subset published by OpenAI in 2024 after human annotators reviewed the original instances and found that a meaningful share were underspecified or effectively unsolvable — issue text that did not describe what the tests required, or tests that depended on the specific implementation the maintainer chose. In other words, the first version of a benchmark built from real repository history contained tasks that no correct solution could pass. That is not a scandal; it is what real issues are like, and it is the clearest possible statement of how much interpretation sits between an issue and a patch.

The rest of the family

The dating trick in LiveCodeBench matters because the SWE-bench repositories are public, old and heavily discussed, so their issues and the patches that fixed them are plausibly in training data. Contamination is not hypothetical here; it is the default assumption for any benchmark built from public repositories more than a year old.

A score belongs to a scaffold, not a model

This is the single most useful thing to understand about coding leaderboards. A SWE-bench submission is not a model answering a question. It is an agent: something that decides which files to open, how to search, what to put in the context, how many attempts to make, whether to run the tests, whether to sample several candidate patches and pick one, and how to format an edit so it applies.

Every one of those choices moves the score, some of them substantially, and none of them are the model. The same model under two scaffolds produces two different numbers; two different models under one scaffold are comparable to each other and to nothing else. So “model X scores Y on SWE-bench” is an underspecified sentence, and the spread attributable to scaffolding is large enough that it can reverse an ordering.

The practical reading: when you compare submissions, check whether they share a scaffold. When a vendor reports a number for its own model in its own harness, that is a measurement of the pair. And when you deploy, your scaffold is neither of theirs, which is the real reason the number does not transfer.

Five ways your repo is not the benchmark

• You have no oracle. SWE-bench tasks come with the tests that define success. Your ticket does not. Most of the difficulty of real work is deciding what “done” means.
• The tasks were selected for having tests. Instances require a PR that changed tests, which filters for exactly the well-specified, testable, self-contained work that models handle best.
• The libraries are public and popular. The model has read the codebase, its documentation, its mailing list and its Stack Overflow tag. Your internal service has none of that.
• The issues are written by contributors. They contain reproductions, versions and expected behaviour. A ticket from your support queue does not.
• There is no deployment, no migration, no rollback. A patch that passes tests is the end of the task, which is roughly the midpoint of a real one.

Building thirty tasks from your own history

You can construct the same benchmark shape from your own repository in an afternoon, and it answers the only question you actually have. The recipe is SWE-bench’s, applied to your merged pull requests:

# Candidate PRs: merged, small, and shipped with a test.
gh pr list --state merged --limit 400 --json number,title,files,mergeCommit \
  --jq '.[] | select((.files | length) <= 3)
        | select(.files | map(.path) | any(test("test|spec")))
        | {number, title, sha: .mergeCommit.oid}'

# For each: base = sha^ ; prompt = the linked issue body (NOT the PR title,
# which usually leaks the solution) ; oracle = the tests the PR added.
git checkout -B task-1471 <sha>^
git checkout <sha> -- tests/            # bring in the oracle, not the fix
pytest tests/test_invoice.py            # must FAIL here, or the task is void

Three details make it honest. Use the issue as the prompt, not the PR description — a PR description written after the fact describes the solution and turns the task into transcription. Verify each task fails at base, which is the check SWE-bench Verified exists because of. And keep the whole thing off the public internet, so it stays uncontaminated for the models you will evaluate next year.

Thirty tasks is enough to see a difference that matters and small enough to build. What it gives you that no leaderboard can: it is your language, your conventions, your test suite, your scaffold, and your definition of done. Why public benchmark rankings fail to transfer generalises the argument past coding.