Evaluating Agents Against What They Are Actually Supposed to Do [Testμ 2026]

The Benchmark Gap

There are no settled metrics for testing AI agents. Plenty of companies have built up a view of what to test and how, and even so there are no shared guidelines to work from.

The consequence Mudit pointed to is the awkward one. Those homegrown suites can still miss critical business and safety failures, returning a passing score while the core rules are being broken.

Traditional benchmarks were built for a different shape of system, and they do not transfer to agentic software. That is the gap Francesca spent the keynote filling.

Her starting position is that evaluation begins with the human loop rather than with a metric. Whether the subject is a machine learning model or a full application, the point of evaluating is to understand the thing you are building well enough to know it serves its purpose.

Evaluation Is Not Reporting

Francesca flagged a habit she sees across teams working in AI and machine learning. Evaluation gets treated as a form of reporting, something performed after the fact to describe what happened.

The reframe matters because it changes the shape of the work. Rather than one end-to-end pass, she described a four-layer process in which each step feeds the next.

Production then feeds back into the start, which makes the whole thing iterative rather than terminal. Nothing about it concludes.

The Four-Layer Evaluation Loop

The loop she presented breaks into four distinct layers, each answering a different question about the agent.

• ASSERT - write down what the system is supposed to do in plain language, and the framework generates built-in evaluators from it.
• Executable tests - self-assertion model graders calibrated against human annotators, plus reference-based scoring wherever ground truth already exists and the work can be automated.
• Domain-specific quality metrics - golden datasets and categories that support metrics particular to your domain, split across code-based, prompt-based, and endpoint-based approaches.
• AI-native experimentation and observability - dashboards and reports that watch live traffic for the failures the tests never anticipated, covering online quality metrics, monitoring, sampled human review, and drift.

She singled out the fourth layer as one of the most important in the journey, despite arriving last. It is the layer that catches what the earlier three could not have predicted.

Case Study: A Support Assistant

To make the layers concrete, Francesca walked through a customer support assistant that answered questions from a knowledge base and could issue a refund through a tool.

It read well, so it shipped. Customers kept calling back, and when the prompt changed nobody on the team could say whether quality had moved in either direction.

The evaluation platform they built around it maps cleanly onto the four layers:

The third layer carries the insight most teams skip. Scoring only the final reply hides an agent that reached an acceptable answer through an unacceptable sequence of steps.

Inside ASSERT

ASSERT stands for Adaptive Specification-driven Scoring for Evaluation and Regression Testing, which is why the acronym does the work. It is open source, and it targets both large language models and agentic systems.

Its input is plain English describing what the agent must and must not do, drawn from product requirements, policy requirements, and system prompts. That written specification becomes the first reviewable artifact in the pipeline.

Francesca used a medical example for the configuration a developer might write: never recommend a specific medication dosage. From there the architecture runs in four steps.

• Taxonomization - the specification is transformed into a behavior taxonomy auto-encoded with permissible and impermissible policy labels, turning each item into a testable pattern rather than a general intention.
• Test set generation - a portfolio of test cases generated from that taxonomy and stratified across test-set dimensions, with single-turn cases as prompts and multi-turn cases as scenarios played out by a simulated tester.
• Inference - the test cases are sent to the target model, application, or agent to produce responses, forming an inference set.
• Judge - the inference set is scored across judge dimensions, with each case marked as passing or flagged at low to high confidence against the policy.

The output is a set of inspectable artifacts carrying row-level scores, surfaced inside the application being built. The chain runs from written specification to behavior categories to assertions to generated test cases to scored conversations.

An assertion, in her definition, is one requirement expressed precisely enough that a single interaction can pass or fail against it. That precision is what separates a specification from an intention.

Baseline Evaluators

The second layer is the standard scorecard every agent receives, and Francesca made the organisational case for it. Without a common set of metrics, there is no way to compare agent quality or performance across teams.

• Groundedness - whether the response is supported by the retrieved context, which she called the first check for any retrieval system.
• Retrieval quality - whether the material the retriever returned is relevant and right for the query.
• Relevance - whether the response addresses the question that was asked, independent of how well it is written.
• Coherence and fluency - the readability measures that sit alongside correctness rather than substituting for it.
• Risk and safety evaluators - the Responsible AI set, comparing against ground-truth answers where they exist and detecting violent, hateful, or unfair content.

An evaluation library runs these locally, including inside a notebook, so a team can read groundedness, retrieval quality, relevance, coherence, and fluency as they work rather than waiting for a pipeline.

Custom Accuracy Metrics

Layer three is where Francesca told the audience to invest time and resources, and it is the layer that cannot be borrowed from another team. Built-in evaluators are shared by design; custom accuracy metrics belong to one agent.

The work starts as a written statement, produced with field experts or the agent’s actual end users, defining what a correct outcome is. She was specific that this happens in collaboration with the product team and business experts rather than inside engineering alone.

Two metrics carried particular weight in her description:

• Task adherence - whether the agent followed the instructions and constraints it was given at the outset.
• Tool call accuracy - whether the correct tool was called with the correct parameters, judged on the trajectory rather than on the final answer.

Custom metrics come in three implementations, and she matched each to the job it suits:

• Code-based - a Python function scoring each item with deterministic logic, well suited to rule checks, keyword matching, and format validation.
• Prompt-based - stronger for subjective quality judgements and semantic similarity analysis.
• Endpoint-based - for custom scoring logic hosted on your own infrastructure.

Her recommendation was to understand all three well enough to combine them, and to run custom accuracy evaluators alongside the built-in set rather than in place of it.

Observability and Tokenomics

The fourth layer runs evaluators against sampled production threads in near real time, which she was careful to distinguish from true real time. The point is that quality gets measured on live traffic rather than only on test sets.

Production failures then get clustered into patterns, and those patterns return to layer one as new assertions and new test cases. That return path is what closes the loop.

Alongside quality, the layer tracks tokenomics: the cost economics of what an organisation spends on AI. Observability tells you how agents are performing, and cost tells you what that performance is worth.

Francesca connected both to business impact, meaning cost and revenue alongside whether the agents are measurably improving team workflows. Combining system health, usage, and quality with cost is what shows whether any of it moved the metric you were trying to improve.

She closed the layer on attribution. Online A/B tests comparing prompt, model, and retrieval changes let improvement be attributed rather than assumed.

Where controlled experiments are not possible, which she noted is often the case in machine learning, offline experimentation with causal inference packages and statistical methods can establish a causal relationship rather than a correlation.

Q & A Session

The keynote ran to time without a live Q&A round, and Francesca invited attendees to bring her specific use cases they wanted to test the framework against. These are the questions the talk sets up and answers.

• Why are generic benchmarks not enough for agents?

  Francesca: Generic metrics are useful signals, but they measure qualities that hold across any system rather than the rules your system was built to respect. An agent can pass them all and still break a policy boundary, because nothing in that scorecard encodes what your agent specifically must and must not do. The specification has to become an input to the evaluation.

• Where should an evaluation effort begin?

  Francesca: With the written behavior specification, before the metrics. Write down in plain language what the system is supposed to do, convert that into behavior categories where each item is a testable pattern rather than a general intention, and generate test cases from there. Evaluation belongs at the beginning of the build rather than after it.

• Should every agent get the same metrics?

  Francesca: Partly. The built-in baseline evaluators should be identical across agents, because that comparability is what lets an organisation judge one agent against another. Custom accuracy metrics are the opposite: each agent needs its own, defined with the business experts and end users who know what a correct outcome looks like in that domain.

Her closing point was that no single layer carries an evaluation practice. It takes open-source packages, observability, production dashboards, built-in evaluators, and custom accuracy evaluators together before an end-to-end AI platform has evaluation worth relying on.

This keynote was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.