You Can’t assertEquals an Agent: A Tester’s Guide to Agentic Quality

Watch the recording

Watch on YouTube ↗

54:11

The first time I tried to write a test for an agentic workflow, I reached for assertEquals — and immediately realized how useless it was. The output was different every run. The tool calls happened in a different order. The reasoning path shifted mid-conversation. Everything I knew about testing said this system was broken. But it was working exactly as designed.

This is not a theoretical problem anymore. Teams across the industry are shipping agents that plan, reason, invoke tools, and make autonomous decisions in production. But testing practices have not kept up — most teams rely on happy-path prompts and manual spot checks because the traditional playbook was never built for systems that think for themselves.

In this talk, I will present a practical testing playbook built from real-world experience of breaking, debugging, and hardening agentic workflows. I will introduce a layered mental model that decomposes any agentic workflow into distinct testable components — giving your team a shared vocabulary to move from “where do we even start” to a structured strategy.

Then I will open a live demo. Using an open-source orchestration framework with a visual graph interface, I will walk through a working agent and systematically break each layer on screen — showing subtle, dangerous failure modes and the concrete testing techniques that catch them. You will walk away with a testing mental model, a working demo you can clone, and the confidence to stop guessing and start testing agents with intention.