HumanEval and the pass@k Metric

10 min read · updated August 4, 2026

HumanEval is 164 Python programming problems, each written by hand for the benchmark, each graded by running unit tests against the model’s completed function. Its headline metric, pass@k, is the probability that at least one of k independently sampled solutions passes — and the estimator used to compute it is not the formula most people assume.

What HumanEval is

HumanEval arrived with OpenAI’s 2021 paper Evaluating Large Language Models Trained on Code. The 164 problems were written by the authors rather than scraped, specifically so that they would not already be sitting in a model’s training data alongside their solutions — the same design instinct behind GPQA, arrived at independently.

Each problem is a single self-contained Python function: a signature, a docstring describing the behaviour, sometimes a doctest-style example, and a hidden set of unit tests. The model is given everything up to and including the docstring and must produce the body. Roughly eight tests per problem on average check the result.

Grading is execution, not similarity. A completion either passes every test or it does not. There is no partial credit, no style score, and no comparison to a reference solution — two completely different correct implementations both pass. That is the single best property of the benchmark, and it is why execution-based grading became the norm for code.

The shape of one problem

A HumanEval item looks like this in structure — the prompt given to the model, then the tests it is graded against.

# --- given to the model (the "prompt") ---
from typing import List

def running_max(numbers: List[int]) -> List[int]:
    """ From a list of integers, generate a list of the running maximum
    element found until the given moment in the sequence.
    >>> running_max([1, 2, 3, 2, 3, 4, 2])
    [1, 2, 3, 3, 3, 4, 4]
    """

# --- the model completes the body ---

# --- the hidden test, run against the completion ---
def check(candidate):
    assert candidate([]) == []
    assert candidate([1, 2, 3, 4]) == [1, 2, 3, 4]
    assert candidate([4, 3, 2, 1]) == [4, 4, 4, 4]
    assert candidate([3, 2, 3, 100, 3]) == [3, 3, 3, 100, 100]

Note what this format contains and what it does not. There are no imports to resolve beyond the standard library, no other files, no existing code to read, and the specification is complete and correct. Every one of those absences matters when the score is read as “coding ability”.

pass@k, worked out

pass@k answers: if I sample k independent solutions and keep any that passes, what fraction of problems do I solve? The honest way to estimate it is to generate more samples than k and compute the expectation combinatorially.

For one problem, generate n samples and count c that pass. The probability that a random subset of size k contains none of the correct ones is the number of all-incorrect subsets divided by the number of subsets. So:

                     C(n - c, k)
pass@k  =  1  -  ---------------
                       C(n, k)

where C(a, b) is "a choose b", and C(a, b) = 0 when a < b.

Then average that per-problem value over all 164 problems.

Two worked cases, on a single problem.

Case A — a problem the model usually gets wrong.
  n = 10 samples, c = 3 pass.

  pass@1  = 1 - C(7,1)/C(10,1)  = 1 -   7/10   = 0.300
  pass@5  = 1 - C(7,5)/C(10,5)  = 1 -  21/252  = 0.917
  pass@10 = 1 - C(7,10)/C(10,10)= 1 -   0/1    = 1.000

Case B — a problem the model almost never gets right.
  n = 10 samples, c = 1 passes.

  pass@1  = 1 - C(9,1)/C(10,1)  = 1 -   9/10   = 0.100
  pass@5  = 1 - C(9,5)/C(10,5)  = 1 - 126/252  = 0.500
  pass@10 = 1 - C(9,10)/C(10,10)= 1 -   0/1    = 1.000

C(7,5) = 21.  C(10,5) = 252.  C(9,5) = 126.
Any problem solved even once in n samples has pass@n = 1 by construction.

Read the last line again, because it is where pass@k gets misused. With k set equal to n, the metric collapses into “did this problem ever get solved”, which is a measure of the sampler’s coverage and not of the model’s reliability. High-k numbers are interesting for research on search and reranking. They are close to irrelevant if you are shipping one generation to a user.

Why the naive formula is wrong

The obvious estimator is to compute the per-sample pass rate p = c/n and then say pass@k = 1 − (1−p)^k. That is biased upward, and the paper says so explicitly, which is why the combinatorial form exists.

Case B again: n = 10, c = 1, so p = 0.1

  Naive:        1 - (1 - 0.1)^5 = 1 - 0.9^5 = 1 - 0.5905 = 0.410
  Unbiased:     1 - C(9,5)/C(10,5)         = 1 - 126/252 = 0.500

Case A: n = 10, c = 3, so p = 0.3

  Naive:        1 - (1 - 0.3)^5 = 1 - 0.7^5 = 1 - 0.1681 = 0.832
  Unbiased:     1 - C(7,5)/C(10,5)         = 1 -  21/252 = 0.917

The gap is several points on a single problem and it does not average away, because the bias has a consistent sign. The naive form treats p as a known constant when it is itself an estimate from n draws; the combinatorial form conditions on what was actually observed. If a paper reports pass@10 from ten samples using the naive formula, it is reporting a different quantity from everyone else.

The estimator is also numerically awkward: computing C(n−c, k) / C(n, k) directly overflows for large n. Implementations compute it as a running product of (1 − k/(n − i)) over the relevant range instead. If you write your own, use the product form.

The temperature that goes with each k

pass@k is a statement about a sampling distribution, so the sampling temperature is part of the metric’s definition rather than an implementation detail. The original paper swept temperature and used different values for different k, because the optimum genuinely moves: low temperature concentrates probability on the single most likely completion, which maximises pass@1, and high temperature spreads it out, which maximises the chance that one of a hundred samples is right.

pass@1 at temperature 0. Common in practice and perfectly defensible, but it is greedy decoding rather than a sample, so the combinatorial estimator does not apply — it is one deterministic attempt per problem. Say which you did.
pass@1 estimated from n samples at a low temperature. A different number from the greedy one, usually slightly lower, and the one the estimator is designed for.
pass@k for larger k at a higher temperature. Report the temperature or the number is uninterpretable.

A pass@1 figure with no temperature attached is missing a field. This is exactly the kind of omission the reporting standard is built to catch.

The tests are weaker than they look

Roughly eight assertions per problem is not much, and they were written by the same people who wrote the problems, so they tend to check the cases the author was thinking about. A solution that is wrong on an empty input, a negative number or a boundary the author did not consider passes anyway.

The EvalPlus work (Liu and colleagues, 2023) is the direct response: it generates a far larger test suite for each HumanEval problem — orders of magnitude more inputs, produced by mutation and by type-aware generation — and republishes the benchmark as HumanEval+. Scores drop on the extended tests, and they drop unevenly across models, which is the interesting part: the ranking is not preserved. A model that scored well partly by producing plausible code that satisfies shallow tests loses more than one that produced code that was actually right.

If you are quoting a HumanEval number in 2026, quote the HumanEval+ number instead, or say which you used. They are different tests.

The siblings, and which to use instead

HumanEval is small and old, and several benchmarks were built to cover what it does not. Knowing which one answers your question saves running the wrong one.

Benchmark	Description
MBPP	Mostly Basic Python Problems: around a thousand short crowd-sourced tasks, each with a description and three assert statements. Easier than HumanEval and larger, so it has a tighter interval — useful for detecting regressions in small models where HumanEval's 164 items are too few. A hand-verified subset exists and is the one worth using.
HumanEval+ / MBPP+	The same problems with vastly extended test suites. Use these rather than the originals whenever you have the choice; they are the same benchmark with the grading fixed.
MultiPL-E	HumanEval and MBPP translated into many programming languages by rule-based transpilation of the tests. The right tool for asking whether a model is weaker in Rust than in Python, with the caveat that a translated problem is not always an equivalent problem — idiomatic solutions differ by language.
BigCodeBench	Tasks requiring calls into real third-party libraries rather than pure standard-library logic, which is much closer to what application code looks like. Harder, and it tests library knowledge alongside reasoning.
Repository-level suites	SWE-bench and its relatives, where the unit is an issue in a real repository rather than a function. A different measurement entirely — see the coding-benchmarks page.

A rough decision rule: use MBPP+ for regression testing small models, HumanEval+ for a quick comparable single number, MultiPL-E when the language matters, BigCodeBench when your code calls libraries, and a repository-level suite when you are evaluating an agent rather than a completion.

What it is a bad proxy for

Working in an existing codebase. No repository, no dependencies, no existing conventions, no code to read first. Issue-resolution benchmarks over real repositories cover that ground — see SWE-bench and friends and agentic benchmarks.
Any language but Python. HumanEval is Python. Ports exist, and translated problems are not equivalent problems — idioms, standard libraries and the natural solution shape all change.
Security, performance or maintainability. The tests check behaviour on a handful of inputs. Code that passes can still be quadratic, injectable or unreadable. There is a fuller treatment in evaluating code generation beyond pass@k.
Contamination-free measurement. Hand-writing the problems protected the 2021 release. Five years of the benchmark sitting on GitHub with solutions has undone that protection. The problems, their canonical solutions and their tests are all public.
Anything specified ambiguously. Every HumanEval docstring is a complete, correct specification. The hardest part of real programming work is that the specification is neither.