SimpleQA Benchmark: How It Scores Guesses and Abstentions

What Is SimpleQA, OpenAI's Benchmark of Short Fact-Seeking Questions?

SimpleQA is an OpenAI benchmark of 4,326 short, fact-seeking questions, introduced in the paper Measuring Short-Form Factuality in Large Language Models (Wei et al., November 2024). Each question has one reference answer, and the questions were "adversarially collected against GPT-4 responses"; at release, GPT-4o and Claude both scored below 50%.

The paper names false outputs "hallucinations" and narrows the problem to questions with a single answer, which makes grading tractable. SimpleQA measures one slice of that wider failure; its causes and other types are explained in the hub on AI hallucination.

The writing rules in Section 2 of the paper explain what kind of fact each question tests:

• One indisputable answer - a question must fix the scope of its answer, asking "which city" or "what year" rather than "where" or "when".
• Answers that do not change - TV, film, video game and sports questions typically name a point in time, and every question had to be answerable as of December 31, 2023.
• Two trainers agree - a second AI trainer answered each question independently, and a question was kept only when both answers matched and their 2 to 4 sources came from two different website domains.
• Hard for GPT-4 - a trainer kept a question only when at least one of four OpenAI model completions got it wrong.
• About 3% label error - a third trainer answered 1,000 sampled questions, and after manual review the authors traced 2.8% of the sample to real data problems such as ambiguous questions and sources that disagree.

There are no answer options. On a multiple-choice test such as the MMLU benchmark, a blind pick earns credit by chance; on SimpleQA a reply earns credit only when it contains the reference answer, and the grader also records when a model abstains. OpenAI's GDPval benchmark grades by comparison instead: experts weigh each model deliverable against a professional's version of the same task, then prefer one or call a tie.

What Kinds of Questions Are in SimpleQA?

A script run on 2026-10-20 read OpenAI's public SimpleQA CSV, which is byte-identical to the MIT-licensed SimpleQA copy on Hugging Face, and counted the topic and answer type stored in each row's metadata. The file holds 4,326 rows, and the counts below are the script's output.

Beside the original, the tables count SimpleQA Verified, the cleaned subset that Google DeepMind and Google Research built from it. Science and technology leads the original set, while Verified trims TV shows and video games:

A score's topic and answer-type mix shapes what it says about a model:

• Dates dominate the original - a model weak at exact dates loses about a third of the original set to one skill, and in the paper's third-trainer check an answer that gave only the year to a month-and-year question counted as incorrect.
• The labels are model-made - OpenAI tagged topics and answer types after the fact with a prompted ChatGPT classifier, so treat a per-topic score as a rough slice.
• The recount matches the paper - the script's ten topic counts equal the topic chart of the SimpleQA paper question for question, and its answer-type shares equal the paper's percentages.

BASE=https://openaipublic.blob.core.windows.net
curl -sLO $BASE/simple-evals/simple_qa_test_set.csv
HF=https://huggingface.co/datasets/google
V=$HF/simpleqa-verified/resolve/main
curl -sLO $V/simpleqa_verified.csv

meta = [ast.literal_eval(r["metadata"]) for r in orig]
topics = Counter(m["topic"] for m in meta)
types = Counter(m["answer_type"] for m in meta)

SimpleQA study, run 2026-10-20
simple_qa_test_set.csv  rows  4326  feee3f7e7db3
simpleqa_verified.csv   rows  1000  b5db21155444

Answer type              SimpleQA      Verified
Date                     1418 32.8%   222 22.2%
Person                   1041 24.1%   198 19.8%
Other                     777 18.0%   249 24.9%
Number                    663 15.3%   185 18.5%
Place                     427  9.9%   146 14.6%

How Is SimpleQA Graded? Correct, Incorrect and Not Attempted

A grader model sees the question, the gold target and the model's reply, and returns one of three letters. OpenAI's reference implementation, simpleqa_eval.py, holds the grader prompt, which says "Hedging and guessing are permissible, provided that the gold target is fully included and the response contains no incorrect information or contradictions." For a gold target of "120k", it counts "124k" as correct, "113k" as incorrect and "around 100k" as not attempted.

The grader is itself an LLM judge, so the usual cautions on LLM-as-a-judge apply: a different grader model or prompt can move a score. The three grades, as the SimpleQA paper defines them in Table 2:

The paper rolls the grades into overall correct, correct given attempted and the F-score, and the reference code computes the same three; the closed form of the F-score is from Appendix B:

overall correct          = c / (c + i + n)
correct given attempted  = c / (c + i)
F-score = harmonic mean of the two
        = 2c / (2c + 2i + n)

• Overall correct - the share of all questions answered correctly, which the paper compares to recall.
• Correct given attempted - the share correct among questions the model answered, which the paper compares to precision.
• Grader check - the authors read 100 correct, 100 incorrect and 100 not attempted grades by hand and found two disagreements with the grader, without a formal study.
• Example from the paper - a model that always answers and gets 30% right scores an F-score of 30%, and a model with 80% correct given attempted needs only 19% overall correct to come close (exactly 30.7%; the paper rounds it to 30%).

Does SimpleQA Reward Guessing?

Yes, under the F-score, and at a lower confidence than the paper's own rule of thumb suggests. Section 2.4 and Appendix B of the SimpleQA paper make the case, crediting Adam Kalai for the point:

• The paper's rule - "if model performance is below 50%, it always makes sense for the model to try to guess if it is at least 50% sure that it will get an answer correct".
• Its proof - Appendix B writes the F-score as 2c / (2c + 2i + n) and shows that one extra guess at even odds has a higher expected score than one extra abstention.
• The exact break-even - working the same algebra for any confidence q, one more guess on a new question scores higher in expectation than abstaining on it when q is above c / (2c + 2i + n + 1), about half of the model's current F-score.

So a model at an F-score of 40% gains, on average, from guessing whenever it is more than about 20% sure. The paper's 50% rule is a sufficient condition; the actual break-even sits lower.

Guess or abstain on the unsure 60% of questions
Base: 35% correct, 5% wrong; q = confidence
Gain from guessing, per 100 questions
  q   accuracy  F-score  penalty 1  penalty 9
 0.1     +6.0     -9.0      -48.0     -480.0
 0.2    +12.0     -3.0      -36.0     -420.0
 0.3    +18.0     +3.0      -24.0     -360.0
 0.4    +24.0     +9.0      -12.0     -300.0
 0.5    +30.0    +15.0       +0.0     -240.0
 0.6    +36.0    +21.0      +12.0     -180.0
 0.7    +42.0    +27.0      +24.0     -120.0
 0.8    +48.0    +33.0      +36.0      -60.0
 0.9    +54.0    +39.0      +48.0       +0.0
Abstain-all F-score 50.0
On 4326 items (1514 right, 216 wrong, 2596 skipped),
one more guess beats abstaining on it
when q > c/(2c+2i+n+1) = 0.2500
Penalty p break-even q = p/(p+1): 0.500 and 0.900

What Changed in SimpleQA Verified?

SimpleQA Verified is a 1,000-question subset of SimpleQA from Google DeepMind and Google Research, described in the technical report SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge (Haas et al., September 2025), which cites noisy and incorrect labels, topical bias and redundancy in the original, such as 119 questions about the founding dates of Colombian municipalities. The report keeps the three grades and the F-score, which it labels F1, and grades with gpt-4.1-2025-04-14, the model OpenAI's current simple-evals code also uses for SimpleQA, but rewrites 88 of the 185 numeric gold answers with an acceptable range and tells the grader to judge only the part of a reply that answers the question and to count a hedge that lists several candidates without committing to one as not attempted. That grader prompt grew from 963 to 1,124 words, which is 16.7% longer, although the report calls it a 15% increase.

Table 2 of the report lists each filter that cut the set down, in order:

The same script matched every row of the MIT-licensed SimpleQA Verified CSV on Hugging Face to its original through the original_index column and compared the two. Its output, run on 2026-10-20:

Verified items against their SimpleQA originals
  gold answer identical           833
  gold answer rewritten            79
    with different numbers in it   18
  numeric answer given a range     88
  question text edited             56
  topic label changed               0
  answer-type label changed        20
  flagged multi_step               73
  flagged requires_reasoning       37

• Most rewrites are formatting - dates move between "June 24th, 1896." and "June 24, 1896" styles; 18 rewritten answers differ in their digits, some only in form, such as "107.0" to "107", and some in substance, such as the date Panama decided to use DVB-T, which moved from "May 12, 2009" to "May 19, 2009".
• Some answers changed outright - the question that asked for the judging-panel member who crowned Reita Faria, Miss World 1966, now asks who crowned her, and its gold answer went from "Lady Annabel Birley" to "Lesley Langley".
• Every range is on a Number item - all 88 acceptable ranges sit on questions typed Number; in the report's own example, a reply of 8,333 against a gold 8,282 moves from incorrect to correct.
• Metadata matches the report - 37 questions flagged as needing reasoning and 73 as multi-step equal the report's 3.7% and 7.3%; topics were not relabelled, while 20 answer types were.

The dataset card sets one rule for running it: no search or retrieval tools. In its words, "With tools, the benchmark is trivial to solve which defeats its purpose."

Why Do SimpleQA Scores Differ Between Leaderboards?

Epoch AI runs SimpleQA Verified itself and, as its SimpleQA Verified methodology page (read on 2026-10-20) states, "We report the simple proportion of questions answered correctly." Since 2026-08-27 it appends an anti-abstention instruction that begins "This is a guessing game, not a test." Asked the bare question, Claude Haiku 4.5 declined on 82% of the 1,000 questions; across the re-evaluated models, declining fell from 15% of questions on average to under 1%.

Check each of these settings before comparing two figures:

• Question set - SimpleQA has 4,326 questions and SimpleQA Verified 1,000 harder, cleaned ones; the two are different tests.
• Formula - the SimpleQA paper and the Verified report headline an F-score, Epoch AI reports accuracy, and OpenAI's reference code returns the share graded correct as its score while only printing the F-score.
• Prompt - an anti-abstention instruction turns abstentions into guesses, which raises accuracy and removes most of the not attempted grades the F-score was designed around.
• Grader - the SimpleQA paper graded with a prompted ChatGPT classifier, OpenAI's current reference code grades with GPT-4.1 and the original prompt, and Verified uses GPT-4.1 with a longer prompt and numeric ranges.
• Model settings - reasoning effort, tools and system prompts differ between runs, and a run with search is not a SimpleQA run at all.
• Contamination - Epoch AI says it suspects Qwen3-Max-Instruct is contaminated on SimpleQA Verified.

The formula alone reorders models. A script run on 2026-10-20 ranked the models in Table 7 of the SimpleQA Verified report three ways, using the report's own percentages:

• Same F-score, different behavior - Claude Opus 4 and Gemini 2.5 Flash score 28.3 and 28.2, yet Opus attempted 35.5% of questions and Flash 96.9%, and Flash answered 8.6 points more questions correctly.
• Penalty favors abstainers - with one point lost per wrong answer, Claude Opus 4 moves from seventh to fourth and GPT 4.1 drops from fourth to seventh.
• The top three hold - Gemini 2.5 Pro, GPT 5 and o3 stay in the top three under every scoring, and all 13 published F-score values match the recomputation.

What Can't SimpleQA Tell You About Your Own Agent?

SimpleQA measures how often a bare model recalls an obscure public fact, and how often it abstains, with no tools. Your support assistant answers from your own documents, so measure its groundedness against a golden dataset of your users' questions; a high SimpleQA score says little about either. Before you build that set, compare the public RAG evaluation datasets by license, size and the skill each one tests.

Once the assistant also opens cases, refunds orders or edits files, the question becomes what it did, which no reply grader observes. TestMu AI's Agent Assurance tests that layer before release. Its command line, Rook CLI, is still pre-alpha, and the public npm package installs it:

npm install -g @testmuai/rook

Builds of the current release cover macOS and Linux, and 64-bit Windows through npm or WSL. Ahead of an npm install, put Node.js 20 or later on the machine; Homebrew and the checksum-verifying shell script cover macOS and Linux and carry their own runtime, and every route is described in the Rook CLI installation guide.

Where most eval and observability tools judge the replies and the traces an agent logged, Agent Assurance looks for what each run changed and lists separately what it could not confirm. Several SimpleQA ideas carry over, with that change in what counts as evidence:

• Gold answer, then acceptance criteria - a SimpleQA item carries one reference string, while Agent Assurance writes scenarios with acceptance criteria from your agent's code, or from its PRD, spec, docs folder or API spec when there is no code to read.
• Not attempted, then Unable to Verify - SimpleQA's third grade records a model that abstained; Unable to Verify marks a criterion the run's evidence could not settle, and such a verdict is left out of the pass-rate denominator.
• Reply grader, then run evidence - SimpleQA's grader sees only the reply. In Agent Assurance, model judges score some criteria, though a message saying the case was filed proves nothing by itself: tool calls are checked against the tools the agent declares, and files, artifacts and records are read, records only through a read-only tool that you sign off on.
• Hallucination as an adversarial category - hallucination is one of the categories in the adversarial class, which a plain generate run plans for by default, next to prompt injection, data exfiltration and PII leakage.

Every run calls the real agent, and nothing it writes during the run is rolled back, so point it at staging. Watching production traffic and ranking foundation models are outside its job: the subject under test is your agent, not the model it calls.

If the assistant only talks to people, over chat, voice, phone, video or image, TestMu AI's Agent Testing fits better: for chat and voice agents its Hallucination Detection metric checks replies against the knowledge base and context you supply.

How to Read a SimpleQA Score Before Picking a Model

Epoch AI's benchmark data download (updated Oct. 19, 2026) shows why margins matter: its top three SimpleQA Verified rows sit within 2.1 points of each other, and each row's 95% margin is wider than that, so the three are a statistical tie. Write down the question set, formula, prompt and grader behind each number you plan to compare, then run through this list:

• Confirm whether the question set is SimpleQA or SimpleQA Verified.
• Confirm whether the formula is an F-score, accuracy or a penalty score, and ask for the not attempted share when only one number is given.
• Confirm the prompt and grader, including any instruction that discourages abstaining.
• Treat gaps smaller than the 95% margin as ties.
• Score your own assistant on your own questions before the model decision is final.

One more check comes after the model is chosen: once your assistant writes to records or files, the Agent Assurance overview explains how Rook CLI derives scenarios for it and where each verdict's evidence comes from.