LLM Evaluation: Metrics, Methods & Tools That Matter in 2026

LLM evaluation measures whether a language model, or the application built on it, produces output that is accurate, grounded, complete, and safe for a defined task. Stanford's 2026 AI Index Responsible AI chapter records hallucination rates from 22% to 94% across 26 top models on one accuracy benchmark, a 72-point spread.^[1]

The metrics section scores four answers to one refund question with five reference-based metrics, and the one answer that gets the refund window wrong beats a correct paraphrase on token F1, BLEU, ROUGE-L, and BERTScore.

For the specific problem of catching unsupported output, our guide to LLM hallucination detection compares groundedness scoring, semantic entropy, judge models and fine-tuned detectors, and sets out the failure each one cannot see.

For the tooling that runs these evaluations, our roundup of LLM evaluation tools compares eleven options on what each measures, where it sits in the lifecycle, and whether it can be self-hosted.

What Is LLM Evaluation?

LLM evaluation is the systematic measurement of language model output against defined quality criteria, producing repeatable scores instead of subjective opinions. It is the language-model case of the broader practice covered in what AI evals are. It answers two separate questions, and the two are easy to confuse.

Model evaluation measures a base model in isolation, typically against public benchmarks, and answers which model to pick. System evaluation measures your application end to end, including the prompt, the retrieval layer, the tool calls, and the guardrails, and answers whether this release is safe to ship.

• Change frequency - Model evals change roughly as often as you change vendors. System evals change every time someone edits a prompt.
• Leaderboard blind spot - A leaderboard score cannot see your retrieval index, so it cannot tell you the answer was grounded in the wrong document.
• Offline vs online - Offline evaluation runs a fixed dataset before release and is the only kind that can block a deploy. Online evaluation samples live traffic after release and catches what the dataset missed.
• Complementary roles - Offline evals tell you whether to ship; online evals tell you what to add to the offline set next.

The distinction matters most for conversational systems, where a single user turn passes through intent handling, retrieval, generation, and policy checks before anyone sees it. Our guide to conversational AI testing walks through how those layers fail independently.

Why Do Public Benchmarks Fall Short?

Public benchmarks fall short for three measurable reasons: invalid questions, fast saturation, and sparse reporting on responsible AI benchmarks. Stanford's 2026 AI Index documents all three. Benchmarks stay useful for shortlisting models and stay unreliable as a release gate; the guide on LLM benchmarks explains what each one measures and how its scores differ from your own evals.

• Validity - The AI Index Technical Performance chapter cites a review that found invalid question rates ranging from 2% on MMLU Math to 42% on GSM8K.^[2] When more than two in five items in a widely quoted test set are broken, small score differences between models are noise.
• Shelf life - The same chapter reports that evaluations intended to be challenging for years are saturated in months, compressing the window in which benchmarks remain useful for tracking progress.^[2]
• Coverage - The AI Index Responsible AI chapter notes that almost all leading frontier model developers report results on capability benchmarks such as MMLU and SWE-bench, while reporting on responsible AI benchmarks remains sparse.^[1]

Use benchmarks to narrow a vendor shortlist to two or three candidates, then build a small task-specific eval set of your own and let that decide the winner. A Testμ 2026 talk on evaluating agents against what they are actually supposed to do takes that further, replacing the benchmark with a written behavior spec.

Which LLM Evaluation Metrics Actually Matter?

LLM metrics fall into three families, sorted by what each one needs before it can compute a score: a reference answer, the input and its retrieved context, or a policy plus adversarial inputs. Choosing the wrong family is why teams end up with dashboards full of numbers that never change when quality does.

The taxonomy below lists the metrics a production system usually tracks, the family each belongs to, how each is computed, and the failure it exists to catch.

Several of these names cover more than one formula. The answer relevance row is the statement count in the DeepEval documentation, while Ragas writes questions from the answer and averages their cosine similarity to the original question. The guide to RAG evaluation metrics explains both and works the statement count, context precision, and context recall by hand on one sample question.

Agents add trajectory and tool-use scores, defined with formulas in AI agent evaluation metrics, and multi-turn assistants add conversation KPIs such as containment rate, collected in chatbot metrics. For code and agent tasks scored over repeated attempts, pass@k and pass^k replace single-run accuracy.

TestMu AI Agent Testing packages one such set for conversational agents: its agent features and metrics documentation lists nine quality metrics that chat and voice evaluations score on a 0 to 100% scale, including bias detection, hallucination detection, completeness, context awareness, response quality, and conversation flow. The guide to chatbot evaluation shows how scores like these become a release decision, with a rubric and a rule written before the run. For a view of those metrics in multi-turn chat and voice sessions, see the conversational AI testing platform page.

ref: Refunds go back to your original payment method
     within 5 business days.
A (right): Your refund goes back to the original
     payment method within five business days.
B (right): We return the money to the card or account
     you paid with in five business days or less.
C (wrong): Refunds go back to your original payment
     method within 10 business days.
D (right): Refunds go back to your original payment
     method within 5 business days

    EM  tokF1  BLEU  ROUGE-L  BERTScore  rescaled
A    0  0.750  28.7    0.640      0.951     0.712
B    0  0.214   4.8    0.200      0.886     0.324
C    0  0.917  76.1    0.917      0.997     0.980
D    1  1.000  92.0    1.000      0.995     0.972

How Do You Evaluate an LLM?

LLM-as-a-judge, human review, and pairwise comparison answer different questions, so pick the method by the decision you need to make. For classifiers, regression models, and other models that do not generate text, turn to AI model evaluation for the metrics and steps that fit them.

A pairwise comparison shows which of two LLM outputs is better, but it cannot show whether either output is good enough to release.

The Agent Testing Certification by TestMu AI ends in a 45-minute exam of 30 multiple-choice questions on the Agent Testing platform, Agent Assurance, and Rook CLI.

How Do You Build an Eval Dataset?

Start with one user journey and 30 to 50 cases sourced from real traffic, then grow the set from production incidents. The dataset decides the value of every metric downstream, because a perfect metric on an unrepresentative set produces confident, wrong verdicts.

• Scope and size - Start with one user journey and 30 to 50 cases. A set that runs in minutes gets run on every commit; a set that takes an hour gets skipped.
• Real-traffic sourcing - Sample actual queries, including the malformed and off-topic ones, rather than inventing tidy questions.
• Permanent incident cases - Each escaped defect becomes a case that runs forever, so the same failure cannot ship twice.
• Distribution coverage - Cover happy paths, edge cases, adversarial inputs, and at least one case per policy rule you claim to enforce.
• Code-coupled versioning - Version the dataset with the application code. An eval set that drifts independently of the prompt it tests produces score changes nobody can explain.
• Held-out slice - Keep one you never optimize against, so you can tell genuine improvement from overfitting to the test set.

Point three is the whole discipline in one line, and it is borrowed directly from conventional QA. The reasoning is the same one behind regression testing: the cheapest bug to catch is the one you have already seen once.

Once a set like this is reviewed, labeled, and versioned, it is a golden dataset, and that guide covers sizing it per slice, versioning it, and catching the decay that sets in as the product changes. For a RAG system, public RAG evaluation datasets offer held-out questions built to test one skill at a time, such as multi-hop reasoning or grounding.

How Do You Run LLM Evals in CI?

Run the eval set as a pipeline gate built from four things: a fixed dataset, a pass threshold, a machine-readable result, and a non-zero exit code on failure. Everything else is reporting.

When the thing under test is the prompt rather than the model, the gate needs a baseline to compare against and a way to catch drift with no local change. Our guide to prompt evaluation covers that narrower case.

A dashboard reports a regression after someone thinks to look at it, while a gate blocks the merge that caused it:

• Trigger - A pull request that changes a prompt, a model setting, or a knowledge source starts the eval job.
• Run - The job runs the fixed eval set against a staging deployment of the application or agent.
• Compare - The job compares the aggregate score with the threshold and with the last known-good run.
• Decide - A non-zero exit code fails the job and blocks the merge, and the machine-readable result stays with the build for triage.

In TestMu AI Agent Testing, the pipeline side is agent-testing-cli, a Python client (Python 3.10 or newer) that starts the Chat and Phone Caller evaluations you configure in TestMu AI. The Agent Testing CLI guide lists every command, and the walkthrough on testing your first AI agent covers the project and suite setup the commands below assume.

A pipeline has no terminal for the interactive login, so set both credential variables from the secret store of your CI platform:

pip install agent-testing-cli
export LT_USERNAME="your-username"
export LT_ACCESS_KEY="your-access-key"

Phone Caller suites can place real phone calls, so the CLI asks for confirmation unless you pass --yes. Add --wait and the CLI polls until the suite finishes; the values for --poll and --timeout are seconds:

agent-testing-cli --project PROJECT_ID run \
  --suite SUITE_ID \
  --yes \
  --wait \
  --poll 5 \
  --timeout 1800

Chat evaluations run asynchronously, so the documentation says not to add --wait to them. The CI form reads the project, workflow, and suite IDs from environment variables, and --json requires --yes:

agent-testing-cli --json \
  --project "$TESTMU_PROJECT_ID" run \
  --workflow "$TESTMU_WORKFLOW_ID" \
  --suite "$TESTMU_SUITE_ID" \
  --yes

That step starts the evaluation without waiting for it, and agent-testing-cli results --suite SUITE_ID shows the suite results once it completes. The CLI documents a fixed set of exit codes, so a job can fail on any value other than 0 without parsing output:

With --wait, a Phone Caller suite that finishes with a failed test returns 1, and one that outlasts --timeout returns 5, so both stop the merge.

Running the full suite on every commit is wasteful, and running nothing is worse. Scale the gate to what changed.

The scheduled row matters more than it looks. An agent degrades without anyone touching it when an upstream model is updated or a knowledge base goes stale, and only a run on a timer catches that class of drift. Agent Testing can schedule chat evaluations on hourly, daily, weekday, weekly, or monthly presets, or on any custom cron expression.

How Do You Stop Eval Gates From Going Flaky?

Gate on an aggregate score across the scenario set instead of individual exact-match assertions, and set the threshold outside your measured run-to-run variance. Non-determinism is a property of the system under test, and the gate has to absorb it.

Skip those two rules and the same commit passes and then fails, engineers re-run the job until it goes green, and the gate stops meaning anything.

This TestMu AI walkthrough shows an autonomous evaluator holding full multi-turn conversations with a deployed agent and scoring each response, which is the repeated sampling that makes an aggregate score meaningful.

• Aggregate scoring - Gate across the scenario set, never on individual exact-match assertions. One unlucky phrasing should not fail a release.
• Repeated sampling - Sample each scenario more than once and use the mean. Single-sample scoring measures the sampler as much as the system.
• Threshold calibration - Measure run-to-run variance on an unchanged build first, then set the threshold outside that band. A threshold tighter than your noise floor is a coin flip.
• Rolling baseline - Compare against the last known-good run rather than an absolute number, so the gate catches deltas instead of re-litigating the target.
• Temperature zero for the judge - Set the judge model's temperature to zero where the API allows it, which cuts scoring variance without touching the system under test. Keep the system under test at its production settings, or the eval measures a configuration users never see.
• Low-volume guards - Weight the verdict by the evidence behind it. The go-live readiness verdict in TestMu AI Agent Testing carries a confidence level set by the number of evaluations behind it: High at 100 or more, Medium at 50 to 99, Low at 20 to 49, and Very Low below 20.
• Quarantine over deletion - A scenario that fails intermittently is a signal about your system, so move it out of the gate and into a tracked list instead of removing it.

The failure pattern is identical to the one QA teams already know from browser suites, and so is the cure: distinguish a genuine regression from noise before anyone stops trusting the signal. Our guide on flaky tests covers the same triage discipline applied to conventional automation.

How Do You Monitor LLM Quality After Release?

Sample live traffic on a fixed schedule and score it with the same metrics the offline gate uses, so a production score and a pre-release score stay directly comparable. Offline evaluation decides whether to ship. Online evaluation tells you what the dataset missed.

• Cost of full coverage - Sample a small fixed percentage of live conversations rather than all of them, because judge-model cost grows with every response you score.
• Baseline alerting - Alert on a drop against a rolling baseline rather than an absolute target. Absolute thresholds fire on traffic-mix changes that are not quality changes.
• Production feedback loop - Route every low-scoring case back into the offline eval set. That loop is what keeps the pre-release gate representative as real usage drifts. In eval-driven development, each routed case must fail on the current version before a fix is written against it.
• Cost and latency - Track them next to quality. A retrieval change that lifts faithfulness while doubling response time is still a regression.

Traces, token counts, and latency come from the observability side of the stack, and the guide to LLM observability covers what to instrument before you start sampling.

Which LLM Evaluation Tools Should You Use?

The tooling market splits into four categories that solve different problems, and a tool from one category rarely does the job of another. The named examples below are representative of each category; confirm current capabilities on each vendor's own site before you commit.

If your evals run on OpenAI's hosted Evals platform, plan the move now. OpenAI's API deprecations page dates the change: existing evals turn read-only on October 31, 2026, with the dashboard and API closing on November 30, 2026.

The first three categories are built for developers: a metric library returns scores and leaves the pass mark to you, and a tracing dashboard records what happened in production. Turning those numbers into a go or no-go call on a release is work your team does on top.

TestMu AI Agent Testing sits in the fourth category. It drives a deployed agent through generated scenarios and personas and returns a Green, Yellow, or Red readiness verdict, which is the shape a release gate needs. It evaluates agents already deployed behind an endpoint or a phone number, so it is not the tool for benchmarking raw base models against each other; for that comparison, a metric library in your own repo is the simpler fit.

Before adopting any of them, check whether it emits a machine-readable report your CI can consume, lets you pin the judge model version, attaches evidence to every verdict, accepts custom criteria alongside the built-in metrics, and runs inside your network when your agent is not publicly reachable. A tool that fails the first check cannot gate anything.

For a category-by-category walkthrough with named options, see our roundup of AI agent evaluation tools.

How Do You Evaluate the Actions of an LLM Agent?

Every metric in this guide reads text: an output, a reference, retrieved context, or a rubric. Once the LLM drives an agent that calls tools, edits files, or updates records, that text is only the agent's report on what it did. A reply such as "Your refund has been issued" can be faithful to the policy, relevant, and complete while the tool log holds no refund call.

Agent Assurance by TestMu AI checks what each run of such an agent changed, before the agent reaches users. It is self-serve with a 14-day free trial, and its command-line client, Rook CLI, installs from npm:

npm install -g @testmuai/rook@latest
rook --version

Rook CLI covers macOS and Linux, and 64-bit Windows through npm or WSL. Installs through npm require Node.js 20 or newer. Homebrew and a checksum-checked shell script are the alternatives, and the Rook CLI installation guide walks through each.

From the agent's source or its specification, Rook CLI derives the scenarios, calls the live agent either against staging or inside a CI pipeline, and scores each criterion against evidence from that run:

• Scenario classes - By default a suite mixes functional and adversarial scenarios, and non-functional scenarios are added on request. The public Rook CLI README maps the adversarial set to MITRE ATLAS, the OWASP Top 10 for Agentic Applications (2026), and the OWASP Top 10 for LLM Applications (2025).
• Evidence per criterion - Files changed on disk during the run, artifacts it produced, the tool calls it made, checked one by one against the tools its definition declares, and read-only queries sent through a tool you have approved.
• Verdicts - Criteria that the evidence settles are marked Pass or Fail, and the rest are marked Unable to Verify, with their own count outside the pass rate. An agent's statement that it acted counts for nothing on its own.
• Reruns - After a second run, the report gives the change in pass rate and lists the scenarios that newly fail or were fixed.

For a CI gate, parse the JSON document that rook run --json prints, where the fields ok, halted, and report.totals report whether the suite finished and how it scored. A finished run can still return exit code 0 with failing scenarios, so the CI walkthrough in the documentation, Run Agent Assurance in CI/CD, gates on those fields.

Agent Assurance is not a general-purpose eval framework or a model benchmark. It tests how your agent behaves, and the metrics earlier in this guide still score the text.

How Do You Know Your Evals Are Any Good?

Measure your evaluator against human labels before you trust it, then re-measure after every judge model change. An uncalibrated eval suite is worse than none, because it produces confidence without accuracy.

• Human-label agreement - Measure it on a fixed sample, and treat that agreement rate as the trust level for the metric. Publish it next to the score.
• Mutation check - Deliberately degrade a response, feed it through the suite, and confirm the score drops. A metric that cannot detect a planted failure is not detecting real ones either.
• Score compression - When every case scores between 0.88 and 0.92, the metric has stopped discriminating and needs a harder rubric or a harder dataset.
• Judge upgrade boundary - Re-measure agreement after any judge model change, and annotate the date on your trend charts so nobody compares scores across the boundary.
• Prevented incidents - Track how many production incidents your suite would have caught. That count measures whether evaluation is working more honestly than the average score does.

A suite whose scores drift upward while incidents keep reaching users is measuring the wrong thing.

Note: Checking an evaluator against human judgment needs the evidence behind each score. Each TestMu AI Agent Testing result carries the multi-turn transcript, a written analysis for every metric, and a pass or fail badge set by the threshold you configured for that metric.

A Starting Plan for Your First LLM Evaluation

Scope your first LLM evaluation to one journey. Pick your highest-traffic user journey and write 30 cases for it this week, then score them on three metrics only: faithfulness, answer relevance, and completeness. Run the set manually once to learn the baseline score and its run-to-run spread.

Then wire it into your pipeline with a threshold set just below that spread, gate pull requests that touch the prompt, and add a nightly scheduled run to catch upstream drift. Expand the set from incidents rather than from planning sessions.

If your system is a customer-facing chat, voice, phone, video, or image agent, TestMu AI Agent Testing scores it with metrics built for that agent type, without an evaluation harness of your own. From a pipeline, agent-testing-cli starts its Chat and Phone Caller suites. The Agent Testing FAQs answer the setup and security questions that come up before a first run.

Where scoring stops and a build gate starts is set out in LLM evaluation vs end-to-end agent testing. Regulated domains change which metrics carry the decision. For a patient-facing deployment, conversational AI in healthcare shows how the published CMS criteria for AI assistants convert into assertions you can run across a scenario set.

Testμ 2026 had a session on exactly this, Eval-First QA Agents: Testing Streaming Platforms at Fox Networks.

Sources and References Used

• [1] 2026 AI Index Report, Responsible AI chapter - Stanford Institute for Human-Centered AI (2026)
• [2] 2026 AI Index Report, Technical Performance chapter - Stanford Institute for Human-Centered AI (2026)
• [3] Bleu: a Method for Automatic Evaluation of Machine Translation - Papineni, Roukos, Ward and Zhu, ACL (2002)
• [4] ROUGE: A Package for Automatic Evaluation of Summaries - Chin-Yew Lin, Text Summarization Branches Out (2004)
• [5] BERTScore: Evaluating Text Generation with BERT - Zhang, Kishore, Wu, Weinberger and Artzi, ICLR (2020)