LLM Benchmarks vs Evals: What Each One Actually Measures

What Are LLM Benchmarks?

LLM benchmarks are standardized test sets that let anyone score a language model on the same task and compare the result to every other model. Each benchmark bundles four things: a fixed dataset, a task format, a scoring rule, and a published number.

The dataset being frozen is the entire point. Because MMLU asks the same questions this year that it asked last year, a 2024 score and a 2026 score sit on the same scale. That comparability is what a leaderboard sells, and it is also the source of every problem covered later in this article.

• Fixed dataset - a question set that does not change between runs, so two models face identical inputs.
• Task format - multiple choice, free-form generation, code patch, or multi-turn tool use. The format decides what capability is actually under test.
• Scoring rule - exact match, unit-test pass, or a judge model. Deterministic rules are cheap and rigid; judge models are flexible and noisy.
• Published score - a single aggregate number, usually reported without the variance, the prompt, or the scaffold that produced it.

That last bullet is where most misreadings start. A leaderboard cell shows one number. Reproducing it requires the prompt template, the sampling temperature, the number of attempts allowed, and the harness version, and those are frequently absent. Note that this is a different activity from benchmark testing in the traditional performance-engineering sense, which measures throughput and latency of a system rather than the capability of a model.

Benchmarks vs Evals

Both produce a score out of 100 and both get called "evaluation," which is why the two are so easily conflated. They answer different questions and belong to different people. A benchmark answers "which model is generally more capable." An eval answers "did this change make our product better or worse." For the anatomy of an eval and the types beyond language models, see what AI evals are.

The practical consequence: a model can win every leaderboard and still fail your eval suite, because your suite measures the prompt you wrote, the documents you retrieved, and the tolerances your users have. Our deeper treatment of the second column, including reference-based and reference-free metrics, lives in the guide to LLM evaluation.

Major LLM Benchmarks and What They Actually Test

Benchmark names get quoted as if they were interchangeable measures of intelligence. They are not. Each was built to probe one narrow capability, and the construction details decide what a score can support.

The GPQA numbers show why construction matters. Its authors report that experts holding or pursuing a PhD in the matching domain reach 65% accuracy on GPQA, while highly skilled non-experts reach only 34% despite spending an average of over 30 minutes per question with unrestricted web access. The 31-point gap is the design: the questions cannot be answered by search, so a model scoring well is doing something other than retrieval.

MMLU works differently. It covers 57 tasks spanning elementary mathematics, US history, computer science, and law, and its authors present it as a test of both breadth and depth of world knowledge. What it does not test is grounding. If your application answers customer questions from a private knowledge base, a strong MMLU score tells you the model knows a great deal in general and nothing about whether it will stick to your documents. For model selection specifically on coding work, our breakdown of the best LLMs for coding maps benchmark families to use cases.

How Benchmark Scoring Works

Three scoring mechanisms cover almost every benchmark you will encounter, and each carries a different amount of hidden noise.

• Exact match - the model picks a letter or emits a string, and the harness compares it to the key. Cheap, fully reproducible, and blind to whether the reasoning behind the answer was sound.
• Executable tests - generated code runs against hidden unit tests, as in HumanEval and SWE-bench. The strongest signal available, because the artifact either works or it does not.
• Judge models - a stronger model grades free-form output against a rubric. This is the only workable option for open-ended tasks, and it introduces the judge's own biases into the score.

Coding benchmarks report pass@k, the probability that at least one of k sampled solutions passes the hidden tests. The gap between one attempt and many is the number worth reading. The original Codex paper reports that the model solves 28.8% of problems on a single attempt and 70.2% when allowed 100 samples, so it could usually find a working solution somewhere in a hundred tries but picked it first fewer than three times in ten. A product that shows the user one answer lives on the single-attempt number; an agent that can compile, test, and retry lives closer to the multi-sample number.

Judge-model scoring deserves particular care, because the grader is itself a language model with preferences for length, formatting, and its own writing style. The mechanics of running one reliably, including calibration against human labels, are covered in our guide to LLM-as-a-judge evaluation.

Where Benchmarks Break Down

Two failure modes degrade every public benchmark over time, and both get worse the more popular the benchmark becomes.

Saturation

A benchmark saturates when the leading models bunch up near the ceiling. The SWE-bench authors reported that at publication the best model they tested, Claude 2, resolved just 1.96% of the 2,294 issues. The Stanford HAI figure quoted at the top of this article puts the Verified subset near 100% today. A benchmark that went from under 2% to near-ceiling has stopped being a ranking instrument, because the remaining spread between models is close to the noise floor and the label error rate.

Contamination

Contamination is benchmark content leaking into training data, so the model recalls the answer instead of deriving it. An open-source contamination report covering more than 15 popular models across six multiple-choice benchmarks measured contamination levels ranging from 1% to 45%, with accuracy gains of up to 14% on contaminated C-Eval and 7% on contaminated HellaSwag. That report also finds larger models gain more advantage than smaller ones on contaminated test sets.

Neither failure mode is detectable from the leaderboard cell. Both are reasons that a benchmark number belongs in the model-selection conversation and not in the release-gate conversation.

What a Benchmark Can and Cannot Gate

The useful question for a QA engineer is not "which benchmark is best" but "what decision is this number allowed to make." A public score is measured on someone else's data, on a model version you may not be running, through a prompt and scaffold you cannot see. That narrows what it can legitimately support.

Rows three and four are the ones teams get wrong. A prompt edit leaves every benchmark score untouched while potentially breaking a third of your production traffic, which is the clearest demonstration that benchmarks and evals are not substitutes.

Build Your Own Eval Suite

An eval suite is a regression suite whose assertions happen to be probabilistic. The workflow will look familiar to anyone who has built a CI test suite, with one structural difference: you gate on a rate, not on a single result.

• Pull 50 to 100 real interactions from production logs, weighted toward complaints, escalations, and the edge cases you already know break.
• Label the expected outcome for each. For open-ended answers, write the criteria a reviewer would apply rather than an exact expected string.
• Pick a scoring method per case, using exact match where the answer is closed, an executable assertion where the output is code or structured data, and a judge model only where neither of those works.
• Run every case several times and record the pass rate, because the same input does not produce the same output twice.
• Set the gate as a threshold on that rate, then wire it to the pipeline exit code so a regression blocks the merge.
• Add every production failure to the set as it is found, so the suite grows toward the traffic that actually breaks.

Step four is the one most teams skip, and it matters even when nothing about the system is probabilistic. To put a number on it, we ran the same deterministic assertion twelve times in a single Chrome session on TestMu AI cloud on 31 August 2026, against the TestMu AI Selenium Playground. The assertion passed all twelve times, and the latency still moved by a factor of nearly five:

run  1   1177 ms  assertion=PASS
run  2    485 ms  assertion=PASS
run  3    438 ms  assertion=PASS
run  4    300 ms  assertion=PASS
run  5    536 ms  assertion=PASS
run  6    455 ms  assertion=PASS
run  7    311 ms  assertion=PASS
run  8    493 ms  assertion=PASS
run  9    247 ms  assertion=PASS
run 10    491 ms  assertion=PASS
run 11    438 ms  assertion=PASS
run 12    513 ms  assertion=PASS

n=12  min=247ms  p50=485ms  p95=1177ms  max=1177ms  mean=490ms
spread max/min = 4.8x   assertion pass rate = 12/12

In our own run above, a gate written against the mean would have accepted the slowest sample as typical. If a fully deterministic check spreads that far across twelve identical runs on the same session, an LLM assertion whose output is sampled will spread further, and the pass rate itself becomes the thing you have to measure. That is why a single-run eval result is closer to an anecdote than a gate, and why per-run variance belongs in your test analytics alongside the pass rate.

For conversational and voice agents, building this by hand means writing the scenarios, the personas, and the scoring rubric yourself. TestMu AI Agent Testing generates 60 to 100 or more test scenarios per workflow directly from your documentation or agent prompt, runs each as a full multi-turn conversation, and scores it on nine quality dimensions including hallucination detection, bias detection, completeness, context awareness, response quality, and conversation flow. Phone agents add more than 30 call-specific metrics on top. The full metric definitions are in the agent features and metrics documentation.

Wiring it into CI follows the same exit-code pattern as any other suite. The command-line client outputs JUnit XML, so GitHub Actions, GitLab CI, Jenkins, and CircleCI all read the results without a plugin:

# Authenticate from CI secrets, never interactively
export TESTMU_USERNAME="${{ secrets.TESTMU_USERNAME }}"
export TESTMU_ACCESS_KEY="${{ secrets.TESTMU_ACCESS_KEY }}"

# Exit code 0 = all scenarios passed, 1 = failures or error
testmu-a2a run --suite agent-regression --format junit --output results.xml

Which to Use When

Map the question you are being asked to the instrument that answers it.

The pattern across every row: benchmarks are a purchasing input and evals are a shipping control. Teams that treat a leaderboard position as a quality gate end up discovering the regression in production. The broader testing strategy for applications built on language models, including where unit tests still apply, is covered in our LLM testing guide.

Where to Start

Open your production logs and pull the fifty most recent interactions where a user rephrased, escalated, or abandoned. That set is your first eval suite, and it will tell you more about your application than any leaderboard.

Label the expected outcome for each case, run every case five times, and record the pass rate rather than the pass or fail. Set your gate below today's rate so the suite starts green, then tighten it as you fix failures. Keep reading benchmarks when you are choosing a model, and stop quoting them once the model is chosen.

If the system under test is a chatbot, voice assistant, or phone agent, TestMu AI can generate that first suite from your existing documentation rather than making you author it by hand, then return a Green, Yellow, or Red production-readiness verdict before launch. The Agent Testing getting-started guide walks through connecting an endpoint and running the first suite.