Beyond Benchmarks: Evaluating Agents Against What They are Actually Supposed to Do

Agent evaluation has moved fast over the past year, bringing trace-level scoring, multi-turn and tool-use testing, LLM-as-judge calibration, and evals wired into CI rather than run once before launch. But one gap keeps showing up: the evaluations teams need for their own system usually don't exist yet. Generic metrics like helpfulness, groundedness, and toxicity are useful signals, yet a system can score well on all of them while still violating the rules that actually matter, such as issuing a refund above threshold, ignoring an approval boundary, or following an instruction hidden in a tool result.

This talk covers where agent evaluation stands today and what has genuinely changed, then focuses on the shift I think matters most: treating your written behavior specification as a first-class input to evaluation rather than background context. I'll walk through ASSERT (Adaptive Spec-driven Scoring for Evaluation and Regression Testing), an open-source framework from Microsoft's Responsible AI team that turns natural-language behavior requirements into executable evals, generating a reviewable behavior taxonomy, stratified test cases, full agent traces, and per-case verdicts with rationale and policy citations. Using a multi-agent travel-planning agent as the worked example, we'll look at where it passes, where it fails, and why an aggregate score would have hidden both.