Build Your Own Benchmark: Why Agents Need Custom Evals (and How to Create Them)

Agents are becoming software’s primary users. The question is no longer which model ranks highest on public leaderboards - it is whether agents can reliably complete the real-world tasks your product promises.

Public benchmarks miss local context: your workflows, state, permissions, edge cases, and tool surfaces. Product failures are specific - bad defaults, missing docs, and friction that only appear when agents actually use your SDK, CLI, or MCP. Custom benchmarks solve this by turning real tasks into reproducible experiments you can rerun after every model, harness, or product change.

In this talk, we’ll discuss how anyone can define their own custom eval suite and benchmarks, what are the advantages of having them, and how to make them scalable and maintainable. You’ll see how to define tasksets, seed environments and agent harnesses, apply treatments (SDK/CLI/MCP/custom agents), add validators, and collect comparable evidence - completion scores, trajectories, metrics, and file changes - so you can compare, debug, and improve systems that work for agents.