Why Agents Need Custom Evals and How to Create Them [Testμ 2026]

Agents As A New Class Of User

Her opening move is that the shift changed who uses a product. Products are no longer used only through their interface, where previously a person worked through documentation, an SDK, a portal or a console.

She dates the change to roughly the past year, starting with coding agents and spreading outward. Her illustration is asking an agent to book a flight rather than going to a travel site yourself.

The new interaction layer she describes on a slide keeps docs, SDKs and products in place, with MCP servers, command line tools and skills sitting on top, and the agent acting through those.

Her evidence that this matters commercially is soft and she does not dress it up: a screenshot of a recent post about companies mentioning MCP usage on earnings calls, plus her own customer conversations putting MCP at maybe a fifth of workflow today. No company is named and no method is given.

The behavioural differences she names are concrete. Agents take error messages literally and try to understand them, where a person would try something else. They read the tool name where a person follows an instruction. And when they hit a deprecated endpoint with an unclear error, they loop, because the human escape hatch of raising a ticket is not available to them.

Eval, Benchmark, Rubric, Trajectory

An eval, in her vocabulary, is a test of one behaviour or a single task, and her example is asking an agent to write an email based on data from a CRM.

A benchmark is a repeatable suite of evals, and she gives the testing-world equivalent explicitly: the equivalent in traditional testing is a regression suite. That is the clearest bridge in the talk for a quality readership.

A rubric is what you grade against, and a trajectory is exactly what the agent did, the steps it took and the decisions it made, meaning the entire log.

Her structural claim is that the task is not just the prompt. On a slide she decomposes it into the instruction, the initial state of the system, the tools, docs and files available, the agent and its harness, and the rubric, all resolving to a final state.

The harness is a first-class variable and she scopes it both ways, covering public harnesses and one you built yourself. That sets up her practical test for anyone claiming to have evals: ask which of those layers they are actually testing.

The Eval That Passed Wrongly

Her definition of a pass is hedged as a position rather than a standard, and she attributes it loosely to research she does not name. A pass depends not only on the final output but on how the agent got there.

Inside a pass she counts the tool calls made, the retries taken, the latency of the surfaces involved, what the agent bypassed, and where it ended up.

The anchor story is first-hand and the database is never named. An experiment tested how well a particular vector database could do a certain thing, and the eval passed. The trajectory showed the agent had decided the database was too hard to work with and had used SQLite instead, completing the same task.

Her conclusion is the sentence the session turns on. Looking at the final output, everything worked, and the whole measurement was wrong because it was looking at the wrong product surface completely.

She also shows a screenshot of an unnamed popular product’s published evals and calls them extremely basic, on the grounds they are questions you could send to a chat model without needing an agent at all, because nothing makes the agent call a tool or touch a real product surface. The company is never named and is not guessed at here.

The Limits Of Public Benchmarks

She lists the public set she has in mind, covering the SWE benches, Terminal-Bench and LiveCodeBench. A fourth name in that list is not recoverable from the captions and is left out.

The question those answer is a ranking question: given a task class, how do ten models compare against each other.

Her point is that most teams no longer have that question, because they already know the task they care about, and she immediately hedges it with an admission that sometimes they might.

She is explicit that public benchmarks earn their place. They create comparability, they let you test the same task across many models, and they give the industry leverage to push model providers to build better models. Reading her as anti-benchmark misses the argument.

Her illustration is a request to set up monitoring for failed payments, which has to traverse permissions, schema, billing state and an email integration, none of which exists in the isolated environments benchmarks run in.

Four Ways Evals Break

On the first, her fix is to state the eval as a job, such as a teammate needing a reusable test payment setup for a demo, rather than as a question with a documented answer.

On the second she names a public benchmark as her example and then defends it in its own context, saying it is not bad for a public benchmark because everyone gets the same instruction and the playing field is fair. Her objection is to that prompt style being reused for internal evals, where it produces a false sense of confidence. That qualifier matters and is easy to drop.

On the third, her example is skills written to one vendor’s guidelines lifting that vendor’s agent while degrading another. The pass-rate figures she attaches are explicitly conditional illustrations of a direction, introduced with an if, and carry no sample, task set, model version or trial count. They are not reproduced here as measurements.

On the fourth she adds a judge problem. The agent writes that it successfully completed the task at the end of the transcript, and a model acting as judge reads that note and concludes everything is fine.

Your Benchmark Is Your Agent UX

She applies the same standard to teams building their own agents, saying that six tool calls to get a simple answer for a user is not good. The number is illustrative.

Her rejection of output-only grading follows: saying the final answer was accurate does not mean it was a real pass or that the agent was efficient enough getting there.

Her definition of a good eval gives the readership a second bridge to familiar vocabulary. A good eval is a simulation of real work in a small unit, almost like a unit test.

Neither analogy is claimed to map perfectly, and she does not push either beyond the comparison.

Five Rules For Writing Evals

The first rule carries a negation that must survive: do not build evals from your API surface. Her precedent is MCP, where teams wrapped entire API surfaces into tools and produced near-duplicates like get user and get customer that look semantically identical to an agent, with no signal about which to call or what identifier encoding to pass.

The second is to write from the user’s job. Rather than instructing an agent to call a named tool and calculate something, write the kind of request a colleague would make, derived from your top ten to twenty user workflows. She pre-empts the objection, noting natural-sounding prompts can still carry enough technical detail to grade.

The third is that rubrics grade output and trajectory together, and must stay fair. A rubric needs to be clear without asking for things the agent was never made aware of.

The fourth is repeated trials, because these are probabilistic systems that will not produce the same output every run, so a single pass rate has no statistical weight. She never gives a trial count.

The fifth is classifying failures into buckets such as a harness problem, missing context, an auth issue or unclear success criteria, because the payoff is product action: docs that confuse agents, deprecated endpoints still sitting in the specification, error messages that need clarifying.

The Adoption Blockers

Her first reason is fear of cannibalisation. A scraping tool or an email provider may not want the agent to become the primary interaction surface, out of concern that people stop coming to the platform at all.

She calls that a very real factor and links it to vendors building their own custom agents rather than exposing MCP servers to popular public ones.

Her prediction is hedged. The industry seems to be moving in a direction where it becomes inevitable that people keep using their own agents, and giving users what they want beats forcing them onto a custom agent you built.

Her one-line diagnosis rests on a negation: agents are still treated as integrations rather than as users.

Her second reason is a direct commercial interest, and the pivot is immediate. There is no good benchmark infrastructure, no good way to measure with that complexity and replicate something equivalent to your actual product, which she names as the reason she built her company. Everything after that point in the main talk is a vendor walkthrough.

The Live Walkthrough

This is a real live screen share of a working web product rather than slides, running roughly four minutes. For one task she opens the instruction sent to the agent, the rubric explaining what the right answer is, the files, described as a mock repository, and the machines that pre-install the dependencies needed to run the code.

She then composes an experiment by picking tasks and choosing which agents to run them against, and actually clicks launch. The interface reports 36 sandboxes created and running in parallel, which is the only number in the session a viewer can watch being produced.

She does not wait for it. Saying she will go to an older run to show results, she switches to a previously completed run, so every result, friction, trajectory and token cost discussed afterwards comes from that earlier run rather than the launch the audience just watched.

The completed run is against Sentry, tested with two coding agents, showing eval output, step count, tool calls, frictions and token cost. Her read is that a query like this can be very slow even on frontier models and consume a lot of tokens.

The finding she highlights is a product-defect claim about a named third party, made from her own tooling with no response from that company. Her account is that a lot of the data coming from that product’s documentation is long enough that it does not fit in the context window, so the agent truncates it, and the engineering team would not know.

The published chapter list claims two things that did not happen. A third company named in a chapter title is not recoverable from the audio, where she starts a word and abandons it, and no results for the second company are ever opened. The side-by-side harness comparison is likewise described as something you could do rather than displayed.

Q & A Session

The host relayed questions from chat, rephrasing and linking them himself, and skipped some for time. No audience member is named anywhere in the recording.

• How do you test an agent’s ability to recover from tool failures or unexpected API errors?

  Haritha Sreedharan Nair: Analyse the trajectory. Most agents still expose their reasoning traces, so you can see where they got stuck and how they tried to recover. Our own product exposes this through an MCP server, so an agent can analyse the trajectory rather than requiring separate analytics. The transferable part of the answer is collecting the traces and making sure you have access to them across different agents.

• Which trajectory details matter, and which data goes unused?

  Haritha Sreedharan Nair: Bucket the run into phases first. Keep tool calls, token consumption per tool call, step duration, and the agent’s immediate response after a tool call, because that reveals whether the tool helped or misled. A lot of the verbosity can be skipped. One coding agent plans at length before answering, while another tries, fails and retries. That last part is a behavioural pattern rather than a measurement.

• How do you handle flaky tool failures or third-party rate limits during long runs?

  Haritha Sreedharan Nair: Run recurringly rather than judging on one trial, and build infrastructure to identify rate-limit responses and pause until slots free up. Parallel benchmark runs can hammer an already exhausted endpoint and cause cascading failures, and the goal is not to make the situation worse. This is the trickiest part.

• What is the best starting point for a solo practitioner or small team?

  Haritha Sreedharan Nair: Write down the use cases, which is equally the hardest part at large companies. Map the user flows, and name the ten things users must be able to accomplish and what success looks like for each. Most people are just lazy to do it. When customers ask us to write their eval sets we can try, but only the customer understands their own users. She names an open framework for building a smaller version yourself, and offers her own product’s free tier.

• How often must custom benchmarks be updated as the product evolves?

  Haritha Sreedharan Nair: The field is early enough that almost nobody has their own benchmarks, so staleness has not come up yet. Speculating: self-maintaining benchmarks might one day flag the tasks a release has invalidated. She declares it a non-answer, flags the speculation as speculation, and adds a personal unsourced estimate about how far most companies are from having benchmarks at all.

• How do you set permission boundaries without exposing real credentials or customer data?

  Haritha Sreedharan Nair: Test accounts, not mocks, because mock answers lose the nuance of the real product. The hard part is filling an isolated account with dummy data representative of real user behaviour, and cleaning up data after each run so evals can be re-run without conflicts is still unsolved. Start with read-only test cases, so runs do not collide.

• How do you weigh an agent completing a task inefficiently against one failing with better reasoning?

  Haritha Sreedharan Nair: I usually ignore the pass or fail and go straight to the frictions, because inefficiency is normally a symptom of friction. From there it splits by what you are testing, and each friction gets marked recovered or unrecovered, the unrecovered ones being worst because the run failed as a result. This is a partial answer: she opens by admitting she is not sure she fully understands the question, and one sentence in the middle breaks off.

• How do you test whether an agent selected the correct tool or API?

  Haritha Sreedharan Nair: Rubrics first: if you know the right tool, add it as a criterion and check whether it was used. Then profile the run by which product surfaces it touched, calling out which server was used and which endpoints were called. That surfaces skills that are never touched by agents and could be deprecated, and the same for unused APIs.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.