Build Trustworthy AI Agents powered by Evals

Agentic AI has captured the industry's attention. As more people experiment with LLMs, they develop intuition aka vibes around whether models perform well. While math and classification are easy to verify, generative tasks aren't. Adding agents makes evaluation far more complex. This talk highlights key challenges in productionizing agentic applications: non-deterministic outputs, elusive ground truth, and models that mysteriously regress. Yet 'LGTM' isn't a deployment strategy.

Unlike chatbots that resemble sophisticated search engines, we expect agents to actually perform tasks; making security, bias, and privacy non-negotiable before unlocking real use cases in healthcare, finance, and legal. We'll explore why agent evaluation is fundamentally harder than traditional ML testing: multi-step reasoning chains, tool-use side effects, and more. Topics include building evaluation datasets reflecting production scenarios, automated LLM-as-judge pipelines, knowing when human-in-the-loop is unavoidable, and detecting regressions before users do.