Evaluating Code Generation Beyond pass@k

pass@k is the standard metric for code generation and it answers a question almost nobody has: if I generate k solutions and a perfect oracle picks the working one, how often is there a working one? In production there is no oracle, and correctness is one of at least five things you care about.

What pass@k estimates

The Codex paper (Chen et al., 2021) defined the estimator that everyone uses. Naively sampling k solutions and reporting whether any passed is a high-variance estimate; instead you generate n > k samples per problem, count the c that pass, and compute the probability that a random size-k subset contains at least one passing solution:

pass@k = E_problems [ 1 - C(n - c, k) / C(n, k) ]

# Numerically stable form, as in the reference implementation:
def pass_at_k(n, c, k):
    """n samples drawn, c of them correct."""
    if n - c < k:
        return 1.0
    p = 1.0
    for i in range(n - c + 1, n + 1):
        p *= 1.0 - k / i
    return 1.0 - p

# 200 samples, 12 correct:
#   pass@1  = 0.060      what one user attempt gets you
#   pass@10 = 0.469      what ten attempts with a perfect picker get you
#   pass@100= 0.998

The spread in that example is the point. The same system is 6% reliable and 99.8% capable depending on which number you quote, and headline comparisons routinely use the larger one.

The oracle hidden in the metric

pass@k for k > 1 assumes something you do not have: a selector that identifies the correct solution among k candidates. In an offline benchmark, the hidden tests are that selector. In your product, the user is — and the user cannot tell, which is the entire problem with generated code.

So the honest reporting rule is: pass@1 is the user-facing number unless you have actually built a selector, in which case report the end-to-end rate of your pipeline including it. Selectors that work in practice are not exotic — run the code, run any tests that exist, typecheck, discard anything that fails — and they convert some of that pass@k headroom into real reliability. But the conversion has to be measured, not assumed.

The complementary metric deserves more use: pass^k, the probability that all k samples pass. Where pass@k measures capability at the top of the distribution, pass^k measures reliability at the bottom, and for anything running unattended — an agent applying a patch, a migration script — reliability is the quantity that decides whether you can ship. A system with pass@1 = 0.9 and pass^5 = 0.4 is a system that will surprise you in production.

Correctness is only as good as the tests

Execution-based grading is the strongest signal available for code, and it inherits the coverage of whatever tests you grade against. Three consequences.

• Shallow suites accept wrong solutions. The EvalPlus effort made this concrete for HumanEval and MBPP by generating substantially more test inputs per problem and showing that a portion of solutions passing the original tests fail the extended ones. If you build your own code eval, generating extra edge-case inputs per problem is the highest-value improvement available.
• Models can special-case the tests. When the test file is visible in the prompt or inferable from the problem, a plausible failure mode is a solution that hardcodes the expected outputs. Detect it by holding out tests the model never sees, and by checking whether the solution references literal values from the test fixtures.
• Passing is not the same as correct on realistic inputs. SWE-bench-style grading uses the repository’s own tests, which were written to cover the human’s fix. A patch that passes them can still be the wrong fix, and this is the structural reason a benchmark score there is a lower bound on “is it right” and an upper bound on “would I merge it”.

Four axes nobody scores

Runtime behaviour

Correct and unusable is a common outcome. Measure wall-clock time against a timeout, and check asymptotics by running the solution against inputs at several sizes and fitting the growth — a solution that is fine at n = 100 and quadratic at n = 100,000 passes every test and takes down a service. Also record peak memory: generated code that materialises an entire result set is a recurring pattern.

Security

Run a static analyser over every generated solution and treat findings as an eval dimension, not a lint warning. A concrete rule set worth having, all of which turn up in generated code with some regularity:

• Shell invocation with interpolated input — shell=True, os.system with an f-string, unquoted arguments.
• SQL assembled by string concatenation rather than parameter binding.
• eval, exec, pickle.loads or YAML full-load applied to input the caller did not produce.
• Disabled certificate verification (verify=False), weak hashes for passwords, hardcoded credentials or keys.
• Path handling that joins user input into a filesystem path with no containment check.
• Silent exception swallowing — a bare except: pass that turns a security failure into a success path.

Maintainability

Cheap proxies that correlate with what reviewers complain about: does it typecheck under a strict configuration; does it pass the project’s existing lint rules; how large is the diff relative to the smallest correct change; does it introduce a new dependency; does it duplicate a helper that already exists in the repository. That last one is worth a specific check — generated code reimplements existing utilities constantly, and a duplication scan against the repository catches it.

Cost of the attempt

Tokens consumed, tool calls made, and wall-clock time to a working solution. A model that reaches the same pass@1 in a third of the turns is a materially better product, and treating cost as an eval axis rather than a footnote is what makes that visible.

Running untrusted code

Every execution-based eval runs model-generated code, which is untrusted code, and the eval harness is frequently the least defended part of a CI system. Non-negotiables:

• A container or microVM per execution, destroyed afterwards. Not a subprocess, and never the CI runner itself.
• Network disabled by default. A generated solution that reaches the internet has either been prompt-injected or is about to make your results unreproducible.
• Read-only filesystem except one scratch directory, with a size cap.
• CPU, memory and wall-clock limits enforced by the runtime, plus a process count limit — fork bombs are a normal output of a sufficiently confused model.
• No credentials in the environment. Not the eval’s API key, not the CI token, nothing. The harness passes code in and reads results out; the sandbox holds no secrets.

This is also why an execution eval belongs on a separate runner from your build. The isolation is the feature, and sharing a runner with anything that holds credentials quietly removes it.