HumanEval Benchmark: 164 Problems, pass@1 and HumanEval+

What Is the HumanEval Benchmark?

The HumanEval benchmark is a set of 164 hand-written Python programming problems that measures whether a code model can write a working function from its signature and docstring. Each completion runs against unit tests, and the headline metric is pass@1, the share of problems solved on the first attempt. The openai_humaneval dataset card on Hugging Face lists the 164 problems as one test split under the MIT license.

OpenAI introduced it in Evaluating Large Language Models Trained on Code by Mark Chen and co-authors, submitted to arXiv on 7 July 2021 to measure the Codex models. The full text of the Codex paper says the tasks "assess language comprehension, reasoning, algorithms, and simple mathematics", and that the problems were written by hand because "our models are trained on a large fraction of GitHub".

• Grading by execution - a completion is correct when the assembled program runs its check function without an assertion error, an exception or a timeout. No text comparison with the reference solution takes place.
• Python only - every problem asks for one Python function, from string handling such as flip_case to small algorithms such as prime_fib.
• Harness status - the openai/human-eval repository opened with "Deprecated and unmaintained" when read on 2026-10-19. Its README still says the execution call in execution.py is commented out, but a January 2025 commit uncommented it, so the harness runs completions as soon as you call it.

HumanEval works differently from knowledge benchmarks such as the MMLU benchmark, where a model picks one lettered option per question. HumanEval executes what the model writes, so any function that passes the tests counts, however it is worded, and any function that slips past weak tests counts too. Fact-recall benchmarks such as SimpleQA differ from both: a grader model compares each short free-text reply with one reference answer.

Fields of a HumanEval Problem: Prompt, Canonical Solution and Tests

Every HumanEval record has five fields, as the Hugging Face dataset viewer shows, and the model sees only one of them. HumanEval/34, unique(), shows the shape, with a check function that holds a single assertion:

# HumanEval/34, prompt (what the model sees)
def unique(l: list):
    """Return sorted unique elements in a list
    >>> unique([5, 3, 5, 2, 3, 3, 9, 0, 123])
    [0, 2, 3, 5, 9, 123]
    """

# canonical_solution (OpenAI's reference body)
    return sorted(list(set(l)))

# test (hidden from the model), line wrapped
def check(candidate):
    assert candidate([5, 3, 5, 2, 3, 3, 9, 0, 123]) \
        == [0, 2, 3, 5, 9, 123]

• Assembly - the harness joins prompt, completion and test into one program and calls check() on the entry point.
• Docstring examples - the examples in the prompt often reappear in the tests, as in unique() above, so a solution can pass by matching the examples alone.
• Uneven test depth - unique() has one test input; HumanEval/53, add(), has 105, because its check function loops over random pairs.

How HumanEval Is Scored: Greedy Runs, Sampling and Temperature

pass@k counts a problem as solved if any of k generated samples passes its tests; the Codex paper, following Kulal et al. (2019), says "a problem is considered solved if any sample passes the unit tests". The Codex paper (PDF) reports pass@1, pass@10 and pass@100, and its fine-tuned Codex-S model solves 37.7% of the problems with a single sample per problem.

The estimator itself, and why a high pass@k can hide an unreliable model, are worked through in pass@k vs pass^k. For reading a HumanEval number, check these settings:

• Greedy pass@1 - one completion per problem, picked deterministically. The EvalPlus leaderboard ranks models this way.
• Sampled pass@1 - the average pass rate over several samples per problem. OpenAI's simple-evals HumanEval script draws 5 per problem.
• Temperature - for a 679M-parameter Codex model, the paper found the best temperature was 0.2 for pass@1 and 0.8 for pass@100, so a pass@1 sampled at a high temperature understates the model.
• Granularity - with 164 problems, each problem is worth about 0.61 points, so a gap of 0.6 points between two models is a single problem and a gap of 1.2 points is two.

How to Run HumanEval on Its Reference Solutions Without a Model

Running the harness on the reference solutions first tests the harness itself, and it needs no model and no API key. The run described here used Python 3.14 with the standard library only, and it executed two kinds of code: the reference solutions shipped with the datasets and hand-written faulty variants. No model output was executed.

# he_lib.py, simplified excerpt:
# run one HumanEval problem in a subprocess
code = (prompt + body + "\n\n" + test
        + "\n\ncheck(" + entry + ")\n")
with open(path, "w", encoding="utf-8") as fh:
    fh.write(code)
proc = subprocess.run(
    [sys.executable, "-s", "-P", path],
    cwd=WORK, env={"PYTHONHASHSEED": "0"},
    capture_output=True, text=True,
    timeout=30)
passed = proc.returncode == 0

HumanEval: 164 problems, 164 with canonical code

Tests per problem   total   mean median  min   max
HumanEval asserts    1181    7.2      7    1    26
HumanEval inputs     1570    9.6      7    1   105
Plus extra inputs  122683  748.1    972    2  1000
HumanEval+ total   124253  757.6    980   12  1100

OpenAI canonical solutions, no model involved
  HumanEval tests      164 / 164 pass
  HumanEval+ inputs    144 / 164 pass
  wrong output or crash: 16 problems
    HumanEval/22    166 of  956 inputs
    HumanEval/44      8 of  478 inputs
    HumanEval/49      1 of  990 inputs
    HumanEval/64      1 of  999 inputs
    (12 more rows in the full output)
  not done in 180 s:     4 problems
    32, 55, 63, 127

Faulty variants, no model involved

ID  Task  Bug                    HE    HE+
M1  9     max starts at 0        pass  FAIL 455/963
M2  29    substring, not prefix  pass  FAIL 33/998
M3  34    result not sorted      pass  FAIL 717/904
M4  35    max starts at 0        pass  FAIL 150/940
M5  31    0 and negatives prime  pass  FAIL 45/167
M6  43    misses a pair of zeros pass  FAIL 120/983
M7  2     parses repr string     pass  FAIL 2/974
M8  58    result not sorted      pass  FAIL 238/989
M9  0     <= instead of <        pass  FAIL 6/1006
M10 52    <= instead of <        FAIL  FAIL 24/425
M11 24    sqrt bound off by one  FAIL  FAIL 7/173
M12 26    dedupes, keeps repeats FAIL  FAIL 840/865
M13 103   rounds half up         FAIL  FAIL 119/951
M14 3     checks final sum only  FAIL  FAIL 199/904
M15 12    returns last longest   FAIL  FAIL 103/936
M16 56    counts brackets only   FAIL  FAIL 55/1012

HumanEval passes, HumanEval+ fails: 9 of 16
Both suites fail:                   7 of 16

HumanEval vs HumanEval+: How Many Tests Does It Take to Catch Wrong Code?

The EvalPlus paper by Jiawei Liu and co-authors, first submitted in May 2023, says "we extend the test-cases of the popular HumanEval benchmark by 80x to build HumanEval+". Across 26 models, the extra tests catch wrong code that HumanEval passed, "reducing the pass@k by up-to 19.3-28.9%".

The counts in the table are the script's output for the v0.1.10 file; the evalplus/humanevalplus dataset card, updated the same day as that release, lists the Apache-2.0 license. The HumanEval figures (mean 9.6, range 1 to 105) match Table 2 of the NeurIPS 2023 version of the EvalPlus paper, which gives a mean of 764.1 for the HumanEval+ build the paper used, a little above the 757.6 counted in v0.1.10.

• Differential testing - in the paper's words, "We use differential testing ... as the oracle to cross-check the output of the ground-truth and LLM-generated solution."
• Input contracts - each HumanEval+ problem carries assertions describing valid inputs, so generated inputs stay inside what the docstring allows.
• Cost of the extra tests - in the v0.1.10 file, the script counted 124,253 HumanEval+ test inputs against 1,570 base inputs taken from HumanEval's own tests, and every model completion has to run through all of them.
• Which to report - quote both. A model whose HumanEval+ score drops far below its HumanEval score solves fewer problems than its HumanEval number suggests.

Published HumanEval and HumanEval+ Scores With Settings and Error Margins

The HumanEval rows below come from the simple-evals README, whose July 2025 note says "simple-evals will no longer be updated for new models or benchmark results". Its HumanEval script asks the model to "fully implement the function described" and averages 5 samples per problem, with no tools enabled for o3 or o4-mini. The Codex-S row is the 2021 baseline from the Codex paper cited in the scoring section.

The HumanEval+ rows come from the EvalPlus leaderboard, which uses HumanEval+ version 0.1.10 and ranks "according to pass@1 using greedy decoding"; the page draws its scores from a results.json data file that, when read on 2026-10-19, was served with a Last-Modified date of 26 December 2024. The five HumanEval+ rows are a selection from the 125 models in that file, not its top five: O1 Mini ties O1 Preview at 89.0%, and GPT-4-Turbo (April 2024) ties DeepSeek-V3 at 86.6%. The same file lists GPT-4o (August 2024) at 92.7% on HumanEval, while simple-evals lists gpt-4o-2024-08-06 at 90.2%, so two harnesses put the same model 2.5 points apart.

The interval column is a 95% Wilson score interval for a proportion over 164 problems, the method the NIST/SEMATECH e-Handbook lists for binomial confidence limits; for the simple-evals rows, which average 5 samples per problem, it is an approximation. A third script, step3_margins.py, computed it; part of its output:

One problem on HumanEval = 0.61 points

pass@1  solved  95% interval (n=164)
 37.7%    61.8   30.6 to  45.3  (-7.1 / +7.6)
 81.7%   134.0   75.1 to  86.9  (-6.6 / +5.2)
 90.2%   147.9   84.7 to  93.9  (-5.5 / +3.7)
 94.5%   155.0   89.9 to  97.1  (-4.6 / +2.6)
 99.3%   162.9   96.5 to  99.9  (-2.8 / +0.6)

• Overlapping intervals - the four HumanEval+ models from 86.6% to 89.0% all overlap, so the EvalPlus leaderboard order among them is not settled by the benchmark.
• Non-integer solves - in the script's output, o4-mini-high's 99.3% comes to 162.9 of 164 problems, and no whole number fits, since 162 solved is 98.8% and 163 is 99.4%. That matches the 5 samples per problem the simple-evals script averages; a greedy score is always a whole number of problems.
• Model picks - for choosing a model for real coding work by task type, see best LLM for coding.

Is HumanEval Saturated, and When Is It Still Worth Running?

Yes, HumanEval is saturated for frontier models: four of the five HumanEval rows in the table above are 90% or more, and the top one leaves about one problem unsolved. Its tests are also public and old, and Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (Riddell, Ni and Cohan, 2024) reports that "18.9% have been seen by models trained on the Stack".

• Frozen reporting - simple-evals stopped adding models in July 2025, and the EvalPlus leaderboard data has not changed since December 2024.
• Coding reports moved on - Anthropic's Claude Opus 4.5 announcement of 24 November 2025 reports SWE-bench Verified and Terminal Bench and does not list HumanEval at all.
• Narrow scope - one self-contained function per problem, with no repository, no files to edit, no tools and no multi-step task. The GDPval benchmark sits at the other extreme, asking for whole work deliverables such as a spreadsheet or legal memo, which occupation experts grade.
• Cross-benchmark view - how repository-level and terminal benchmarks score coding work, and how to read any benchmark number, is covered in LLM benchmarks.

HumanEval still earns a run in a few places, as long as HumanEval+ runs beside it:

• Small and local models - in the EvalPlus results file on GitHub, models of 1B to 2B parameters score from 2.4% (zyte-1B) to 65.9% (DeepSeek-Coder-1.3B-instruct) greedy pass@1 on HumanEval, far below the ceiling and well spread out.
• Harness checks - running the canonical solutions confirms a new evaluation pipeline in minutes, before any model output runs.
• Before and after a change - quantizing or fine-tuning a small model can change which problems pass, and pairing the two runs problem by problem shows which ones.
• Teaching - the five-field format is the simplest way to show how execution-based grading works.

Beyond HumanEval's Unit Tests: Checking a Coding Agent's File Changes and Tool Calls

A HumanEval pass means one function returned the expected value for the inputs someone wrote down, and the faulty variants above show how little that can be. A coding agent does more than return a value: it edits files, runs commands, calls tools and then summarizes its own work, and a green test run is silent about a file written outside the task or a forbidden tool that got called anyway.

For a coding agent, TestMu AI's Agent Assurance checks, before release, what a HumanEval check function never sees: what the run changed in the repository and which tools it touched on the way. The Rook CLI that runs these checks is at the pre-alpha stage, and two commands install it from npm and print its version:

npm install -g @testmuai/rook
rook --version

Supported systems are macOS and Linux, and 64-bit Windows through npm or WSL, and the npm package expects Node.js 20 or newer. To run Rook CLI from inside Claude Code, the Claude Code setup guide adds the skill installer, which needs Node.js 22 or newer.

• Scenarios read from the repository - Rook CLI reads the coding agent's source or its spec, including declared tools, subagents and MCP configuration, and writes scenarios with acceptance criteria; the default mix is functional plus adversarial, with non-functional checks added only on request.
• Changed files as evidence - every file a run changed under the paths its profile declares is recorded, so a write that falls inside those paths but outside what the task asked for shows up in the evidence.
• Tool calls against the agent's definition - each call observed in the run is compared with the tools listed in the agent's own definition, and a scenario can list tools the run must never call.
• A report of passing tests is not evidence - a coding agent's own statement that every test passed decides no criterion, and model judges that score some criteria never count a claimed action as proof; the Agent Assurance overview docs explain why the agent's own reply carries the least weight.
• Three verdicts per criterion - a criterion such as "the changed module gained a test file" is marked Pass, Fail or Unable to Verify, and results in the third group are reported on their own and do not count toward the pass rate.

Give the agent a disposable clone of the repository or a CI runner you can throw away, because whatever it writes during a run stays written. Most eval and observability tools judge an agent by what it replied and logged; Agent Assurance looks at what a run changed in files and tool calls, and names the criteria it had no way to check.

Checklist Before Comparing Two HumanEval Scores

Before you compare two HumanEval numbers, write down the suite, the setting and the source of each, then work through these checks:

• Suite and version - HumanEval or HumanEval+, and which HumanEval+ release.
• Setting - greedy or sampled pass@1, how many samples, which prompt and which harness.
• Interval - put the 95% interval for 164 problems next to each score; gaps inside it are ties.
• Your own tasks - score shortlisted models on tasks from your codebase, kept in a golden dataset that never enters a prompt.
• Coding agents - if the model will drive a coding agent, test the agent's file changes and tool calls before release; the Agent Assurance CI/CD guide shows how to run Rook CLI in a pipeline.