MMLU Benchmark: 57 Subjects, Label Errors and Score Margins

What Is the MMLU Benchmark?

The MMLU benchmark, short for Massive Multitask Language Understanding, is a test of a language model's knowledge across 57 subjects, which the cais/mmlu dataset card lists by name. Each question has four answer options and one keyed answer, and the score is the share of test questions the model answers with the keyed letter.

Dan Hendrycks and six co-authors introduced it in Measuring Massive Multitask Language Understanding, submitted to arXiv on 7 September 2020 and published at ICLR 2021. In the paper, the 175-billion-parameter GPT-3 reached 43.9% few-shot, and the authors estimated expert-level accuracy at about 89.8%.

The full text of the MMLU paper states that the test set has 14,079 questions, while the cais/mmlu dataset on Hugging Face (MIT license), which evaluation harnesses download, has 14,042 test rows. Both counts are correct for their source.

• Quote the count your harness used - a score computed on cais/mmlu is a share of 14,042 questions, so write that number beside it.
• Keep the test split out of prompts - the dev split exists so few-shot examples never come from the questions being scored.

MMLU's 57 Subjects and How Many Questions Each One Has

MMLU's subjects are uneven in size. The category map in the original MMLU repository sorts the 57 subjects into STEM, humanities, social sciences and other, and the counts below apply it to the 14,042 test rows.

MMLU test split, cais/mmlu (read 2026-10-19)
  test rows        14,042
  validation rows  1,531
  dev rows         285 (5 per subject)
  subjects         57
  per subject      min 100, median 173, max 1534
  under 150 items  22 subjects

Questions by category
  STEM             18 subj    3018  21.5%
  Other            14 subj    3242  23.1%
  Social sciences  12 subj    3077  21.9%
  Humanities       13 subj    4705  33.5%

// mmlu-audit.mjs (abridged). The full script pages
// through cais/mmlu and mmlu-redux-2.0 on the
// Hugging Face datasets-server, 100 rows a request.
const API = "https://datasets-server.huggingface.co";

// 95% Wilson interval, half-width in points
function wilsonHalf(p, n, z = 1.96) {
  const d = 1 + (z * z) / n;
  const s = (p * (1 - p)) / n + (z * z) / (4 * n * n);
  return (100 * z / d) * Math.sqrt(s);
}

// Test questions per subject and per answer letter
for (const r of rows) {
  bySubject[r.subject] = (bySubject[r.subject] || 0) + 1;
  letters[r.answer]++;
}

// Redux 2.0: anything but "ok" or "expert" is an error
for (const r of reduxRows) {
  const t = r.error_type;
  if (t !== "ok" && t !== "expert") err++;
}

How MMLU Is Scored: Zero-Shot, Five-Shot and the 25% Guessing Floor

In the MMLU paper and its evaluation script, each prompt opens with "The following are multiple choice questions (with answers) about" the subject, followed by the question and its options A to D. The five-shot setting adds five solved questions from that subject's dev split first; the zero-shot setting adds none.

• Log-likelihood scoring - the lm-evaluation-harness MMLU template compares the model's probability for the letters A, B, C and D after "Answer:" and counts the most likely letter, taking few-shot examples from the first rows of the dev split.
• Generated answers - chat models are often scored on the answer they write, sometimes after reasoning step by step, which is the zero-shot chain-of-thought setting OpenAI's simple-evals uses.
• Aggregation - the harness's MMLU group weights each subject by its size, so the headline number is the share of all test questions answered correctly.

Four options put the random-guess floor at 25%, but the answer letters in the test split are not spread evenly. Output of the same script, run on 2026-10-19:

Answer key letters (test split)
  A   3222  22.9%
  B   3462  24.7%
  C   3582  25.5%
  D   3776  26.9%
  always-same-letter floor: 26.9%

• Always-D baseline: 26.9% - a model that always answers D scores 1.9 points above the 25% chance average with no knowledge at all.
• Letter bias shows up as a score - a model that leans toward D on questions it cannot answer gains a little, and one that leans toward A loses a little, so compare small models against 26.9%, not 25%. On SimpleQA, by contrast, each reply is graded correct, incorrect or not attempted, so a wrong guess costs more than declining.

How Accurate Are MMLU's Answer Keys?

The MMLU-Redux paper reports that 14 human experts re-annotated 5,700 questions, 100 from each subject, against a taxonomy of errors in the question and in the key. Its authors write that scores on the corrected questions were "notably altered, changing their ranking" of the models they tested.

The MMLU-Redux 2.0 dataset, linked above the subject table, is released under CC BY 4.0 and stores one error_type label per question. Output of the same script, run on 2026-10-19:

MMLU-Redux 2.0 annotations (5,700 questions)
  ok                         5330  93.5%
  bad_question_clarity        132  2.3%
  wrong_groundtruth           106  1.9%
  multiple_correct_answers     39  0.7%
  no_correct_answer            36  0.6%
  expert                       32  0.6%
  bad_options_clarity          25  0.4%
  flagged, any type            370  6.5%
  errors (not ok, not expert)  338  5.9%
  key cannot be right          142  2.5%

Highest error share by subject (excl. expert)
  virology                     57%
  logical_fallacies            26%
  college_chemistry            25%
  professional_law             18%
  human_sexuality              17%
  business_ethics              14%
  formal_logic                 13%
  global_facts                 12%
  subjects with 0 errors     10
  subjects at 10% or more    12

• Errors cluster in a few subjects - Virology has 57 errors per 100 checked questions and 10 subjects have none, so a model's weak subject may be a weak key.
• The ceiling sits below 100% - 142 of the 5,700 re-checked questions (2.5%) have a wrong key or no correct option, so answering them correctly cannot earn those points against the original key.
• Check misses against the labels - before reading a low subject score as a gap in knowledge, count how many of the misses fall on questions Redux flags.

MMLU Margins of Error: One Subject Against the Whole Test

Each question is one pass-or-fail trial, so the uncertainty in a score depends on how many questions stand behind it. The script uses the Wilson interval, which the NIST Engineering Statistics Handbook describes as the method introduced by Wilson in 1927, because it stays inside 0% to 100% for small subjects and scores near the top.

Output of the same script, run on 2026-10-19:

95% margin, Wilson (points)   at 70%   at 90%
  abstract_algebra (100)       +/-8.8   +/-6.0
  median subject (173)         +/-6.8   +/-4.5
  professional_law (1534)      +/-2.3   +/-1.5
  whole test (14042)           +/-0.8   +/-0.5

• One small subject is a rough reading - at 70% on a 100-question subject, the 95% interval runs from about 60% to 78%, so a 5-point subject gap between two models proves little.
• The whole test is tight - 14,042 questions bring the margin down to about 0.8 points at 70% and 0.5 at 90%, which makes a one-point gap worth a second look and a tenth of a point meaningless.
• Margins cover sampling only - they leave out prompt wording, letter bias and key errors, which can move a score more than sampling does.

Two models scored on MMLU answer the same questions, so a fairer comparison counts the questions on which they disagree and tests that split with a paired test, as worked through in the golden dataset guide. That needs per-question results, which is one more reason to log samples when you run the benchmark.

MMLU vs MMLU-Pro vs MMLU-Redux vs MMMLU

The MMLU-Pro paper describes "expanding the choice set from four to ten options" and removing trivial and noisy MMLU questions; it was accepted at NeurIPS 2024 in the Datasets and Benchmarks track. The MMLU-Pro dataset holds 12,032 test and 70 validation questions in 14 disciplines, drawn from MMLU, STEM problem sites, TheoremQA and SciBench.

OpenAI's MMMLU dataset card says the MMLU test set was translated "into 14 languages using professional human translators". Its dataset viewer lists 14,042 questions per language, 196,588 in all, when read on 2026-10-19.

All four ask a model to pick one option from a list. A benchmark such as GDPval instead grades finished work deliverables against the same work done by professionals, which is closer to what an agent delivers.

Published MMLU Benchmark and MMLU-Pro Scores With Their Margins

The MMLU rows come from OpenAI's simple-evals repository, which emphasizes "the zero-shot, chain-of-thought setting" and has not been updated for new models since July 2025. The MMLU-Pro rows come from Table 6 of the DeepSeek-V3 technical report, which names simple-evals prompts for MMLU but does not name MMLU-Pro's prompt or shot count, and caps output length at 8K for every model.

• Margins - each margin is the 95% Wilson half-width at the published score, computed with the wilsonHalf function from the script shown earlier, on 14,042 questions for MMLU and 12,032 for MMLU-Pro, in a run on 2026-10-19.
• Read across one source only - the two blocks use different runners and settings, so an MMLU number and an MMLU-Pro number never compare directly, and neither does a number from another lab's card.

Is MMLU Saturated?

Yes, for frontier models. The top MMLU score in the table, 93.3%, is above the 89.8% expert estimate in the MMLU paper (PDF), and in the Redux 2.0 sample 2.5% of questions cannot be answered correctly against the original key, which leaves little room above it.

• Gaps inside the margin - in the DeepSeek-V3 report's chat model table, Llama 3.1 405B scores 88.6%, DeepSeek-V3 88.5% and Claude 3.5 Sonnet 88.3% on MMLU, a 0.3-point spread inside a margin of about 0.5 points.
• Rankings that flip - in the same table of the DeepSeek-V3 report (PDF), the three score 86.2%, 89.1% and 88.9% on MMLU-Redux, so the first-place model on MMLU comes third on the re-checked questions. The report ran MMLU-Redux zero-shot in a different prompt format, so the flip mixes corrected keys with a change of format.
• MMLU-Pro still separates them - the four MMLU-Pro scores in the table above span 5.4 points, from 72.6% to 78.0%, while MMLU put three of those models within 0.3 points.

MMLU still serves two jobs: placing small and mid-size models, which score well below the ceiling, and comparing a new model with years of older published scores. Saturation tables for other benchmarks, contamination checks and the general error-bar method are in the LLM benchmarks guide.

How to Run MMLU With lm-evaluation-harness

EleutherAI's lm-evaluation-harness ships the mmlu task, which loads cais/mmlu; its latest release on GitHub was v0.4.13 on 31 August 2026 when read on 2026-10-19. A five-shot run of a Hugging Face model:

pip install "lm_eval[hf]"
lm_eval --model hf \
  --model_args pretrained=<org>/<model> \
  --tasks mmlu \
  --num_fewshot 5 \
  --batch_size auto \
  --output_path results \
  --log_samples

• Pin the harness - record the harness version and the task version printed in the results, because prompt templates change between releases.
• Pick the setting on purpose - set num_fewshot to 5 to match five-shot reports, or to 0 to match zero-shot ones, and write it next to the score.
• Use the right endpoint - the harness's API guide says multiple-choice tasks "such as MMLU" run only on completion endpoints, since they need log-probabilities. For a chat-only API, run mmlu_generative, which grades the letter the model writes.
• Report margins - give the whole-test score with its margin from the table above, and subject scores with their own, wider margins.
• Audit the misses - join the logged samples with MMLU-Redux 2.0 labels by subject and question, and count how many misses sit on flagged questions.

For a model you plan to deploy, MMLU only narrows the shortlist. The steps in AI model evaluation cover scoring the shortlisted models on tasks from your own product.

Agents Built on High-MMLU Models: Testing What the Agent Does

An MMLU score is a property of the model: how often it picks the keyed letter on an exam question. An agent built on that model is judged on other outcomes, such as whether the refund it reports exists in the billing system, or whether it called a tool it was told never to call.

TestMu AI's Agent Assurance tests the agent itself before release. Rook CLI, its command-line product, is pre-alpha and publicly installable; the npm route is one command:

npm install -g @testmuai/rook

Rook CLI ships for macOS and Linux, and 64-bit Windows through npm or WSL; on the npm route it depends on Node.js 20 or newer, and two routes for macOS and Linux only, Homebrew and an install script that verifies checksums, are documented in the Rook CLI installation guide.

• Suite built from your spec - Rook CLI reads your agent's source, a product requirements document, a documentation folder or an API spec and generates scenarios, attaching acceptance criteria to each; functional and adversarial classes are in its default mix.
• Live calls against staging - scenarios reach your agent through an HTTP endpoint, a command or an MCP tool, and nothing the agent writes during a run is rolled back, so point Rook CLI at a staging or CI environment rather than production.
• Proof from the run - criteria are decided from the state a run produces: file changes within the paths declared for the agent, artifacts it produced, every observed tool call checked against its declared tool list, and records fetched through a read-only tool that you approve. Some criteria are scored by a model judge, yet a claim the agent makes about its own actions never decides one.
• Pass, Fail, Unable to Verify - each criterion gets one of these verdicts, and those that could not be checked stay out of the pass rate, which therefore counts observed outcomes only.

Most eval and observability tools score what your agent said and recorded. For an agent built on a high-MMLU model, Agent Assurance checks what each run changed instead and lists what it could not verify; it does not benchmark models, so an MMLU number can still inform which model to try while Agent Assurance tests the agent you build on it.

Reading an MMLU Score Before a Model Decision

Start by putting the two MMLU benchmark numbers side by side with their settings and sources: if those differ, the comparison stops there. If they match, a whole-test gap under about one point, or a subject gap under that subject's margin in the table above, counts as a tie, and your own tasks decide.

When the model you pick will power an agent with write access to files, tickets or databases, the Agent Assurance overview shows how to run Rook CLI against a staging deployment of it before release.