Model Evaluation for QA Engineers: ML Metrics in QA Terms

What Is Model Evaluation?

The mechanics are four steps. Hold back labelled examples the model never saw in training, run the model across them, compare each output to its label, and compute scores from the counts of right and wrong answers.

None of that is foreign to QA. The output shape is. A test run answers "did this break?" and model evaluation answers "how often, and in which direction, is this wrong?"

Standards bodies treat both as one discipline. The NIST AI Risk Management Framework defines its MEASURE function as putting in place "objective, repeatable, or scalable test, evaluation, verification, and validation (TEVV) processes including metrics, methods, and methodologies". Test and evaluation sit in the same acronym for a reason.

NIST also states that measurement processes "should include rigorous software testing and performance assessment methodologies with associated measures of uncertainty, comparisons to performance benchmarks, and formalized reporting and documentation of results". Rigorous software testing is the phrase in that sentence a QA engineer owns.

Capgemini's World Quality Report 2025 found that 50% of respondents report their organizations lack AI/ML expertise, unchanged from 2024. Not all of that gap needs years of retraining to close. A large part of it is vocabulary.

ML Evaluation Terms in QA Language

Each row below pairs an ML term with the testing concept it most resembles, then names the place the two stop being the same thing. The third column is the important one: it is where teams get burned assuming the analogy holds.

Two rows deserve extra attention. The threshold row explains why evaluation work stalls: nobody owns the number. The LLM-as-judge row explains why evaluation results are sometimes wrong with no visible symptom, and it is the row that most often surprises teams migrating from testing AI applications by hand.

Why Does Model Evaluation Have No Pass or Fail?

A functional test encodes its own verdict. An assertEquals(200, response.status) call carries the standard inside the assertion, so the test tells you whether it passed.

Model evaluation returns 0.72. Nothing in the number says whether 0.72 ships. Somebody has to decide, and that decision is a product decision informed by what each kind of error costs.

Here is a real run. The script below scored a 28-case evaluation set for a support agent, where the positive class is "should escalate to a human", using the formulas in the next section. This is the actual console output:

$ node score-eval-set.mjs
cases scored      : 28
confusion matrix  : tp=9 fp=4 fn=3 tn=12
accuracy          : 0.750
precision         : 0.692
recall            : 0.750
f1                : 0.720
gate recall>=0.90 : FAIL
missed escalations: ESC-03, ESC-06, ESC-10
$ echo $?
1

Accuracy of 0.750 reads as tolerable. Recall of 0.750 means three of the twelve cases that needed a human never got one, and the report names them: ESC-03, ESC-06 and ESC-10. Same run, same data, opposite conclusions depending on which number the team agreed to read.

The practical move is to write the threshold down before the run, next to the reason. Three sentences are enough:

• Name the positive class and say which error costs more. Here, a missed escalation costs more than an unnecessary one.
• Pick the metric that punishes that error. A missed positive lowers recall, so recall is the gate.
• Set the number and the owner. Recall at or above 0.90, owned by the support quality lead, reviewed every quarter.

TestMu AI's Agent Testing platform makes this explicit rather than tribal. It scores chat and voice agents across nine quality dimensions including hallucination detection, bias detection, completeness, context awareness and root-cause understanding, then lets you set a minimum acceptable score per metric and configure whether higher or lower is better for each one. The threshold stops being a number in somebody's head.

Named threshold configurations matter more than they look. A "Strict" config for a healthcare flow and a "Default" config for a marketing FAQ bot let one team run one evaluation set at two standards, which is the thing a shared global threshold cannot do.

The Metrics You Will Actually Be Handed

Most metrics come from four counts. True positives and true negatives are correct answers; false positives are false alarms; false negatives are misses. The scikit-learn precision_score reference defines precision as the ratio tp / (tp + fp), and its companion recall_score as tp / (tp + fn).

F1 is the balanced combination of the two, the special case of the F-beta measure where beta equals 1 and precision and recall are weighted equally. Read the table by the last column, which names the failure each metric conceals.

Generative systems add metrics that have no confusion matrix behind them, because the output is text rather than a class. Hallucination rate, completeness and context awareness are scored by a judge, human or model, against the source material. Those are the dimensions the Agent Testing metrics documentation details for chat, voice, phone and image agents.

A judged metric carries a second model's error on top of the first model's. Treat the judge as a component under test: sample its verdicts, have a human relabel that sample, and track the agreement between them as its own number on the report.

How to Build an Evaluation Set Like a Test Suite

Test design skills transfer directly here. Equivalence partitioning, boundary analysis and negative testing are exactly how you decide which examples belong in an evaluation set.

What changes is proportion. A test suite covers each partition once because one case proves the branch works. An evaluation set needs enough examples per partition for the score to mean something, since a partition with three examples produces a metric that swings by a third when one prediction changes.

• Partition by user intent, not by feature - group the set the way a user arrives, so "refund request", "plan change" and "outage report" each get their own score alongside the headline number.
• Size each partition for stability - a partition where one flipped prediction moves the metric more than the margin you gate on is too small to make a release decision from.
• Label the boundary cases first - the ambiguous examples are where model versions actually differ, and a set made only of clear cases scores high on every version and distinguishes nothing.
• Keep adversarial phrasing in the set - angry, terse and misspelled inputs belong in the evaluation data, because an agent that handles polished phrasing is not the same as an agent that handles users.
• Freeze a subset permanently - a locked group of examples that must never regress gives you a stable reference across model versions, in the way a smoke suite does across builds.
• Record who labelled each example - when two labellers disagree, the argument is about the label, not the model, and you can only see that if labeller identity is stored.

Leakage is the one failure mode with no QA equivalent. If an example in the evaluation set also appeared in training data, the model has seen the answer, and every metric computed on it is inflated. Version the evaluation set separately from training data and keep the split reproducible.

Note: TestMu AI's Agent Testing generates evaluation scenarios directly from an uploaded PRD or requirements document, then runs them across personas and adversarial inputs so the set covers phrasing a hand-written suite misses. Start evaluating your agent free.

Running Evaluation in CI and Gating the Release

Evaluation is a pipeline job like any other. It scores a held-out set, compares each metric to its threshold, and exits non-zero when a threshold is breached, which is the contract every CI platform already understands.

The scoring script from earlier already follows that contract: it returned exit code 1 when recall fell below 0.90. A hosted evaluation follows the same shape, and wiring one into GitHub Actions needs no plugin.

name: Agent Evaluation

on: [push]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install the TestMu AI agent testing CLI
        run: pip install testmu-a2a-cli

      - name: Score the held-out evaluation set
        env:
          TESTMU_USERNAME: ${{ secrets.TESTMU_USERNAME }}
          TESTMU_ACCESS_KEY: ${{ secrets.TESTMU_ACCESS_KEY }}
        run: testmu-a2a run --format junit --output results.xml

      - name: Publish the run as a build artifact
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: evaluation-results
          path: results.xml

This behaves like the rest of your suite for concrete reasons. Authentication comes from environment variables rather than an interactive login, JUnit XML is read natively by GitHub Actions, GitLab CI, Jenkins and CircleCI, and exit code 0 or 1 wires straight to pipeline pass or fail.

Keep the results file as an artifact on every run, including failures. A metric drop tells you nothing on its own, so the per-example results from the previous run are what turn "recall fell 4 points" into a list of examples to look at.

The pattern that holds up uses two gates. A blocking gate on the metric tied to the expensive error, and a reporting-only gate on the rest, so a 2-point wobble in a secondary dimension does not train the team to ignore red builds. Aggregated trends across those runs are what test intelligence dashboards are for.

Regression Testing a Model That Is Not Deterministic

Regression testing assumes a stable baseline. Run the suite on the old build, run it on the new one, and any case that flips from green to red is the regression.

Generative models break that assumption at the first step, because the same prompt produces different text on two consecutive calls. Diffing the strings produces a wall of differences with no defects in it, which is the trap teams fall into when they first apply testing techniques for non-deterministic AI outputs to a model upgrade.

Score behaviour instead of text. Everything below rests on comparing distributions rather than individual outputs:

• Compare distributions, not strings - run the same evaluation set against both versions and compare the aggregate metrics, treating a drop beyond an agreed margin as the regression signal.
• Set the margin from observed variance - run the current model against the set several times first, and set the margin wider than the spread you measure, otherwise run-to-run noise fails builds on its own.
• Segment before you conclude - a flat headline metric routinely hides one partition getting better while another gets worse, so compare per-partition scores before signing off a version.
• Guard the frozen subset absolutely - the locked examples get a zero-tolerance gate, which gives you at least one check with the binary behaviour a regression suite is supposed to have.

Model upgrades are the moment this pays. A provider version bump can raise the headline score and still break the one intent your highest-value customers use, and only per-partition comparison shows it. TestMu AI's Agent Testing runs this comparison as scheduled regression coverage after model updates, alongside pre-launch validation, so the check happens on a cron rather than when somebody remembers.

Public benchmark scores are not a substitute for this. A benchmark measures someone else's task on data your users never send, so a model that climbs it can still regress on the single intent your revenue depends on. Deeper treatments of the judged-metric side live in our guides to LLM evaluation and RAG testing.

Conclusion

Start by writing down one threshold. Pick the model your team already ships, name the error that costs the most, choose the metric that punishes it, and put a number and an owner next to it before the next evaluation run. That single artifact converts a score nobody acts on into a gate.

Then move the run into the pipeline where the rest of your suite lives, keep the per-example results as artifacts, and gate on the one metric that matters rather than all of them. The translation work is most of the job, and the QA skills underneath it already transfer.

To score a live agent against these dimensions, the Agent Testing platform covers chat, voice, phone and image agents with per-metric thresholds and CI-ready output, and the testing your first AI agent guide walks through the first run end to end. If you are also evaluating tooling, our roundup of AI agent evaluation tools compares the options.