What Are AI Evals? How They Work and How to Run One

What Are AI Evals?

An AI eval is a scored test for an AI system: a fixed dataset of inputs, grading logic applied to every output, and an aggregate score tracked across releases. The word is shorthand for evaluation, and it stretches from a ten-case script a developer runs locally to a scheduled suite scoring thousands of production conversations.

The reason evals exist as a separate practice is that AI output is non-deterministic. Ask a support agent the same question twice and you get two different sentences, both correct. An equality assertion fails on the second one even though nothing regressed, so the assertion has to be replaced with something that scores degree of correctness.

Swapping the boolean for a score changes three things downstream. A score has no natural pass mark, so somebody has to choose a threshold and own it. One case proves almost nothing on its own, which moves the unit of measurement to the aggregate across a dataset. Scores are also only meaningful next to earlier scores, making evals comparative by design: you are always measuring this version against the last one.

What Are the Four Parts of an Eval?

Every eval, from a scratch script to a managed platform, is built from the same four parts. If one is missing, what you have is a demo.

• The dataset - the inputs you run through the system, plus whatever reference material the grader needs. Real support tickets, real user prompts, and real past failures beat invented examples, because invented examples encode what you already believe the system does.
• The task - the thing that turns one input into one output. For a chatbot that is a single reply; for an agent it is a full multi-step run, including tool calls.
• The grader - the logic that scores that output. OpenAI's graders documentation describes them as comparing reference answers to model-generated answers and returning a grade in the range from 0 to 1, and lists string checks, text similarity, model-scored grading, and Python code execution as the available types.
• The threshold - the aggregate score that decides whether the change ships. This is the part teams skip, and it is the only part that turns a dashboard into a gate.

Graders split into two families with different costs. Code-based graders are cheap, instant, and completely deterministic, but they only catch what you can express as a rule: a required phrase, a forbidden phrase, valid JSON, a number in range. Model-based graders handle judgment calls like tone and helpfulness, but they cost tokens, add latency, and need their own validation against human labels. Most working eval suites use code graders for the mechanical checks and reserve model graders for the subjective dimensions, a split covered in depth in our guide to LLM-as-a-judge evaluation.

How Are Evals Different From Software Tests?

If you already own a test suite, most of an eval maps onto something you have. The mapping is close enough to be useful and different enough to matter.

The last row is where most QA teams get stuck. In a deterministic suite, a test that passes and fails on identical input is broken and gets quarantined. In an eval, that variation is the measurement itself, and quarantining it removes exactly the signal you built the eval to capture. The practical answer is to run the same case several times and score the distribution rather than a single sample.

What Types of AI Evals Are There?

Evals are usually grouped by what is under test, because that decides what the dataset contains and what the grader can see. The four groups below cover most production work, and a single application often needs three of them.

Are Evals the Same as Benchmarks?

Benchmarks and evals both score model output, which is why the words get swapped. They answer different questions. A benchmark scores general capability against a public dataset that everyone shares, so results are comparable across labs. An eval scores your application against your data, and the results mean nothing to anyone else.

The consequence for a release decision is direct: a public benchmark score cannot tell you whether a model upgrade will break your refund policy responses, because the benchmark has never seen your refund policy. Benchmarks narrow the shortlist of models worth trying; evals decide which one ships. Our breakdown of LLM benchmarks versus evals covers what each one can and cannot gate.

How Do You Run Your First Eval?

A first eval does not need a framework. It needs a dataset, a grader, and a threshold, and it can be a single script. The steps below produce something that runs in CI on day one and grows from there.

• Collect 20 to 50 real cases - pull them from support transcripts, logged prompts, and the bug tickets where the system already embarrassed you. That range is a practical starting point, not a rule anyone published.
• Write down what correct means - per case, as required phrases, forbidden phrases, or a rubric a human could apply consistently. If two people on the team disagree about a case, the grader cannot score it either.
• Start with code graders - substring checks and format validation cover more ground than teams expect, run instantly, and cost nothing. Add a model grader only for the dimensions a rule genuinely cannot express.
• Record a baseline - run the current production prompt through the set and save the score. Every future run is compared against this number, so an eval without a baseline is just a dashboard.
• Exit non-zero below the threshold - this is what makes the script a gate instead of a report. The docs for testing your first AI agent walk through the same loop on the TestMu AI platform.

OpenAI's evaluation best practices guidance is to ensure eval data includes typical cases, edge cases, and adversarial cases.

Here is a working harness at that scale. It scores ten recorded support-agent responses with a code-based grader, compares a candidate prompt against a saved baseline, and exits non-zero when the aggregate drops below 0.80.

const THRESHOLD = 0.80;

// One row: the input, what the answer must contain, and what it must never contain.
const cases = [
  { id: 'no-invented-date',
    mustInclude: ['check', 'Billing'],
    mustNotInclude: ['15th', 'January'] },
  { id: 'multi-turn-recall',
    mustInclude: ['A-4417'],
    mustNotInclude: ['what is your account number'] },
  // ...20 to 50 rows pulled from real transcripts
];

function grade(output, c) {
  const text = output.toLowerCase();
  const missing = c.mustInclude.filter((t) => !text.includes(t.toLowerCase()));
  const leaked = c.mustNotInclude.filter((t) => text.includes(t.toLowerCase()));
  const checks = c.mustInclude.length + c.mustNotInclude.length;
  // Partial credit, not a boolean: this is what makes it an eval.
  const score = checks === 0 ? 1 : (checks - missing.length - leaked.length) / checks;
  return { score, missing, leaked };
}

const passRate = cases.reduce((sum, c) => sum + grade(c.candidate, c).score, 0) / cases.length;

if (passRate < THRESHOLD) {
  console.log('EVAL GATE FAILED');
  process.exit(1);   // the line that turns a report into a gate
}

Running that harness against a candidate prompt that introduced a hallucinated renewal date produced this output:

baseline (v1)   pass_rate=1.00
candidate (v2)  pass_rate=0.80   threshold=0.80

REGRESSED  no-invented-date   missing=["check"]  leaked=["15th"]
REGRESSED  out-of-scope       leaked=["better than"]
REGRESSED  multi-turn-recall  missing=["A-4417"]  leaked=["what is your account number"]

EVAL GATE PASSED

Read the last line carefully, because that run is a working demonstration of the most common eval bug. Three cases regressed, including an invented renewal date and a lost account number, and the gate still reported a pass. The seven untouched cases scored a perfect 1.00 each, which pulled the mean up to exactly 0.80, and 0.80 is not below a threshold of 0.80.

How Do You Gate a Release on Evals?

The run above shows why a single aggregate threshold is not enough on its own. A mean is a lossy summary, and adding easy cases to a dataset raises it, which means an eval suite can get less sensitive as it grows. Layering gates catches what the mean alone lets through.

• Aggregate floor - the mean score across the dataset must clear the threshold. This catches broad quality collapse and nothing subtler.
• Per-case regression check - no case that scored 1.00 on the baseline may drop below it. This is the gate that would have caught all three regressions above, and it is one comparison against saved baseline scores.
• Critical-case subset - a tagged group where any failure blocks the release regardless of the mean. Safety refusals, compliance language, and data-disclosure cases belong here, because averaging them against routine cases is how a serious failure gets absorbed.

Trigger the suite on every prompt edit, model version bump, and retrieval index rebuild, since all three change behaviour without touching application code and none of them fire a normal test. OpenAI's guidance calls this continuous evaluation and recommends running evals on every change rather than only ahead of a launch. For metric selection and CI wiring at production scale, our guide to LLM evaluation metrics and methods goes deeper than this article does.

Where Do AI Evals Go Wrong?

Most failing eval programs fail the same handful of ways, and none of them are about picking the wrong metric.

• The dataset is invented - cases written by the person who wrote the prompt test what the author already thinks the system does. Real traffic contains the phrasings nobody anticipated, and those are the ones that break things.
• The dataset silently changes - adding or editing rows makes today's score incomparable with last month's, and the trend line everyone is watching becomes fiction. Version the dataset and record which version produced each score.
• The judge is never validated - a model grader is itself an AI system with failure modes. Label a sample by hand, measure how often the judge agrees, and re-check that agreement whenever the judge model changes.
• Nothing is gated - a suite that reports without blocking gets ignored within weeks. The threshold and the non-zero exit are what give the numbers consequences.
• The eval set never grows - every production incident is a case the dataset was missing. Teams that add each incident to the set get a suite that gets sharper over time; teams that do not keep re-learning the same failure.

Automated scoring also has a hard ceiling: it only catches failure modes somebody already wrote a grader for. Reading a sample of real transcripts each week is what surfaces the modes nobody anticipated, and each one found that way becomes a new row in the dataset. Teams tracking behaviour after release should pair this with LLM observability, which captures the production traces the weekly sample is drawn from.

Where Should You Start With AI Evals?

Open your last three production incidents and turn each into three dataset rows: the exact input that failed, a paraphrase of it, and the neighbouring case you suspect is broken too. Give every row its required and forbidden phrases. That is a nine-case eval built entirely from failures you have already paid for, and it runs as a script this afternoon. Add the baseline comparison and the non-zero exit, and it is a release gate by the end of the week.

When the agent under test has multiple turns, tool calls, or a voice channel, a homegrown harness stops paying for itself. TestMu AI Agent Testing covers that surface with scored dimensions and confidence levels out of the box, and the Agent Testing platform documentation walks through connecting an agent and reading its first results. Teams testing LLM features inside a broader QA process can start from our LLM testing learning hub.