OpenAI GDPval Benchmark: Tasks, Grading, Scores and Limits

What Is GDPval, OpenAI's Benchmark of Real Work Deliverables?

The GDPval benchmark is OpenAI's test of how well AI models produce real work deliverables, such as a spreadsheet, a legal memo or a slide deck, for 44 occupations in the 9 US sectors that each contribute more than 5% of GDP, as set out in OpenAI's GDPval report. Occupation experts grade a model's file against the file a professional made for the same request.

The GDPval paper was submitted to arXiv on 5 October 2025 by Tejal Patwardhan and colleagues at OpenAI. Task writers needed at least 4 years of experience in their occupation, and each task got an average of five human reviews, never fewer than three.

The times come from the paper's summary tables for each set. A 5-hour median puts GDPval tasks far from the single-question format of most model benchmarks: each one is a short work assignment with attachments. The MMLU benchmark, built from multiple-choice exam questions, and SimpleQA, built from short fact-seeking questions, are typical of that format.

GDPval Gold Subset: Sectors, Files and Rubric Items Across 220 Open Tasks

The gold subset lives in the openai/gdpval dataset on Hugging Face. Its commit history there shows the first upload on 25 September 2025 and a commit titled "Release GDPval v2 (rubrics + deliverables)" on 10 February 2026, which added the rubric and expert-deliverable columns counted below.

Tasks: 220
Sectors: 9, occupations: 44
Tasks per occupation: 5

Tasks per sector
   25  Professional, Scientific, and Technical Services
   25  Government
   25  Information
   25  Manufacturing
   25  Real Estate and Rental and Leasing
   25  Finance and Insurance
   25  Wholesale Trade
   25  Health Care and Social Assistance
   20  Retail Trade

Reference files
  tasks with at least one   125 (56.8%)
  files in total            261
  most in one task          17
  top types  xlsx 86, pdf 74, docx 67, png 9, wav 8, zip 3

Expert (gold) deliverable files
  files in total            248
  top types  pdf 85, xlsx 65, docx 64, pptx 17, zip 5

Rubric items (rubric_json)
  items in total            10453
  per task min/median/max   14 / 47 / 137
  written by a human        10453
  penalty items (score < 0) 94 in 18 tasks
  median points available   71.5
  items naming a file/format 1614 (15.4%)

Prompt length (words)
  min/median/max            100 / 307 / 1064

GDPval Grading: Expert Pairwise Comparisons and the Win-or-Tie Rate

For the gold subset, experts from the task's occupation saw the request, the reference files and two or more unlabeled deliverables, then ranked them with written reasons. According to the paper, grading one comparison took over an hour on average.

• Three outcomes - each comparison scores 1 when the model's deliverable is preferred, 0 when the professional's is, and 0.5 for a tie.
• Win rate and win-or-tie rate - results can count wins alone or wins plus ties, and the paper's headline figure counts both.
• Repeats - each model was sampled 3 times per prompt and each sample was graded by 3 different experts, giving 9 comparisons per prompt per model.
• Grader agreement - on the paper's agreement measure, where a tie set against a win counts as half agreement, two experts grading the same sample scored 70.8%, and OpenAI's experimental automated grader, based on GPT-5-high, scored 65.7% against experts.
• Ungradable tasks - 12 of the 220 tasks were marked ungradable for the automated grader, which had no internet access and could run only Python.

Blinding was partial. The paper notes that OpenAI outputs often used em dashes, Claude outputs often wrote in the first person and Grok sometimes named itself, so graders may have guessed the source.

OpenAI's GDPval grading page now says the rubrics and gold deliverables are open-sourced, so you run your own grading. It calls pairwise human expert preference the standard and an LLM judge on the rubrics "a rough estimate", and its leaderboard page says the OpenAI-hosted leaderboard "is no longer active".

Published GDPval and GDPval-AA Scores

The first four rows come from Table 2 of the GDPval paper and use expert graders on the gold subset. The last three come from the Artificial Analysis GDPval-AA leaderboard, version 2.1, read on 2026-10-19, and give each Elo rating with its 95% interval.

• One metric in the paper rows - all four figures are the win rate the paper's Table 2 uses for its speed and cost analysis, counting only deliverables rated better than the human one; the paper plots its win-or-tie results per model in a chart with no matching table, so they are not repeated here.
• No conversion - a percentage against a professional and an Elo rating against other models are different scales, so no row in the top half can be compared with a row in the bottom half.
• Dated rows - the Elo rows change as models are added and the fit is updated; check the leaderboard before quoting them.

GDPval vs GDPval-AA: Expert Graders Against an LLM Judge and Elo

GDPval-AA runs OpenAI's open tasks through Artificial Analysis's own harness. Its page says models get "shell access and web browsing capabilities in an agentic loop via Stirrup", and its GDPval-AA methodology says a judge sampled from a panel of three frontier LLMs ranks each pair of submissions blindly, with ties counted as half-wins in the Elo fit.

Setup and grading together can move a score a long way. The inspect_evals GDPval README from the UK AI Security Institute reports that GPT-5 with low thinking scored 47.3% ± 6.2% through OpenAI's automated grader, and notes that OpenAI's official leaderboard reported a lower 31.9% for wins only and 34.6% for wins and ties. The README does not attribute the gap, and its run swapped OpenAI's code interpreter for bash and Python tools in a Docker sandbox.

• Self-preference - the paper found its GPT-5-based grader agreed less with experts when grading capable OpenAI models, which the paper links to a 2024 study finding that LLM evaluators favor their own generations.
• Judge choice - an LLM judge is cheaper than an hour of expert time per comparison, and the trade-off is the agreement gap above; LLM-as-a-judge covers how to calibrate one against human labels.
• Harness - a shell, a browser and installed libraries decide which file types a model can open and produce, so the same model can score differently in two harnesses.

How Big a GDPval Score Gap Is Real?

The script below treats each task as one independent trial and computes the 95% margin of a win-or-tie rate on the 220 gold tasks and on the 1,320-task full set.

// gdpval-margin.mjs: 95% margin of a win-or-tie rate
// One task = one independent trial (normal approx.)
const z = 1.96;
const margin = (p, n) =>
  100 * z * Math.sqrt((p * (1 - p)) / n);
const log = (s) => console.log(s);

log("95% margin (percentage points)");
log("rate   220 tasks   1,320 tasks");
for (const p of [0.3, 0.4, 0.5]) {
  const a = margin(p, 220).toFixed(1);
  const b = margin(p, 1320).toFixed(1);
  const pct = (p * 100).toFixed(0);
  log(`${pct}%      +/-${a}       +/-${b}`);
}
log("");
// Two independent scores on the same set: variances add
const gap = (p, n) => Math.SQRT2 * margin(p, n);
log("Gap needed between two models near 40%");
log(`  220 tasks:   ${gap(0.4, 220).toFixed(1)} points`);
log(`  1,320 tasks: ${gap(0.4, 1320).toFixed(1)} points`);

Run with Node.js 25.5 on 2026-10-19, it printed:

95% margin (percentage points)
rate   220 tasks   1,320 tasks
30%      +/-6.1       +/-2.5
40%      +/-6.5       +/-2.6
50%      +/-6.6       +/-2.7

Gap needed between two models near 40%
  220 tasks:   9.2 points
  1,320 tasks: 3.7 points

• 220 tasks - a 50% rate is good to about ±6.6 points, close to the ±6.2 that the inspect_evals run reports from bootstrapping its grades.
• Two models - near 40%, two independent scores on the gold subset need about 9 points between them before the difference stands out from sampling error; on the full set, about 4.
• Repeats do not add tasks - the paper's 9 comparisons per prompt narrow grader and sampling noise, but the margin on task difficulty still rests on 220 tasks.

Both models answer the same 220 tasks, so a paired comparison is usually tighter than this independent-samples gap; the score-gap section of the LLM benchmarks reference covers the paired and clustered methods for any benchmark.

What Does GDPval Not Measure?

The limitations section of the GDPval paper's HTML version names most of the gaps itself. Others follow from the counts above.

• Physical and interpersonal work - manual labor is excluded, and so are tasks needing extensive tacit knowledge, personal data, proprietary software or communication between people.
• Finding the context - tasks are fully specified and one-shot. In the paper's under-contextualized experiment, prompts cut to 42% of their token length made GPT-5 perform worse.
• Long tasks - win rates were highest on tasks of 0 to 2 hours and fell as expert completion time rose.
• Severity - a loss is a loss in the score; when experts re-rated GPT-5's losing deliverables, about 29% were bad or catastrophic and about 3% catastrophic.
• How the work was done - graders see the request, the reference files and the deliverables, and every rubric item describes the files, so commands run, sources fetched and tools called are not part of the grade.
• Your own work - five tasks per occupation say little about the documents, policies and tools of one team; a golden dataset built from your own cases answers that.

The process gap is small while a model answers in one turn. With a shell and a browser, as in GDPval-AA, a model can download data, run code and write files on the way to its deliverable, and none of those steps reaches the judge.

How to Run GDPval Yourself

The inspect_evals GDPval implementation runs on the UK AI Security Institute's Inspect framework, and its README on GitHub gives the install and run commands:

pip install "inspect-evals[gdpval]"
inspect eval inspect_evals/gdpval \
  --model openai/gpt-5-nano

The same README says the task runs in a Docker sandbox and that building its image can take up to 10 minutes, so build it before the first run; the README gives the build command. Its last step, submitting results through OpenAI's grading form, no longer applies: OpenAI's grading page now asks you to run your own grading, as described above.

• Fix the setup first - record the model version, reasoning effort, tools, harness and installed libraries; the paper's own prompt tuning added 5 points to GPT-5's win rate.
• Run all 220 tasks, several times - the paper sampled each model 3 times per prompt, and dropping hard tasks changes what the score means.
• Collect every deliverable file - the inspect_evals task writes them to a timestamped output folder that can be uploaded to Hugging Face.
• Grade with experts where the decision matters - pairwise expert preference is the standard OpenAI names; an LLM judge on the released rubrics and gold deliverables is the rough estimate.
• Report the margin - give n, samples per task, the grader and the 95% margin next to the score, and say whether it counts wins or wins and ties.

OpenAI's automated grader could not grade 12 of the 220 tasks, so a run graded by an LLM judge should list the tasks it skipped.

Agents on GDPval-Style Work: Checking the Actions Behind an Agent's Deliverable

A GDPval-AA run is already an agent at work: shell access, web browsing and files written along the way. When your own agent does work like this for customers, the file it hands back is one output, and the records it changed, the tools it called and the files it touched are the rest.

Before such an agent reaches customers, you can put it through release tests with TestMu AI's Agent Assurance. The product ships as the Rook CLI, which is pre-alpha and publicly installable:

npm install -g @testmuai/rook
rook --version

It runs on macOS and Linux, and 64-bit Windows through npm or WSL; npm needs Node.js 20 or newer, and Homebrew and the shell installer are covered in the Rook CLI installation guide.

• Scenarios from your agent - it reads the agent's code, its spec or an API spec and writes scenarios with acceptance criteria, functional and adversarial by default.
• Staging runs - your actual agent is called against a staging environment or from CI, and anything it writes there stays written, because nothing is rolled back.
• Artifacts - for an agent that returns a report or file, it checks that the artifact exists, its type and size, and its content where it can be read; what it cannot read is Unable to Verify.
• Actions - it compares tool calls with the tools the agent declares, including ones it must not call, checks files changed under the declared paths and reads records through a read-only tool you approve.

Each criterion ends as Pass, Fail or Unable to Verify. Model judges score some criteria, yet an agent saying it saved the file or updated the record is never accepted as proof; the evidence from the run decides, as the Agent Assurance overview docs describe.

Most eval and observability tools score what your agent said and recorded. Agent Assurance checks what the run changed, and reports what it could not verify. It is not a model benchmark: it tests your agent, not the model underneath, so it complements a GDPval number in a model choice.

Using GDPval in a Model Decision

Start with the occupations closest to your work, read the five gold tasks for each, and treat a gap smaller than the margin worked out above as a tie. Then score the shortlisted models on your own cases, using the steps in AI agent evaluation.

If the model will run inside an agent that changes files or records, install the Rook CLI with npm install -g @testmuai/rook and point Agent Assurance at a staging copy before the agent ships.