Salesforce's Four Phases of Agentic Testing [Testμ 2026]

Many Valid Paths

Conventional software is a tried and true path with a known input and a known output. When it breaks, a single component or feature failed, and you can trace back to where things went wrong.

Agentic systems break that in every direction. There are many valid paths, things can go wrong at different turns, outcomes span a wide range, and a customer journey walks through multiple workflows before it resolves.

The consequence he leads with is not a metric. When an agent fails, customers lose trust in the agent and in the company behind it.

Phase 1: Validating Every Step

In 2024, as generative AI features started shipping, the focus was on the output itself: whether what the AI generated was accurate, and whether accuracy, conciseness and completeness could be measured at all.

The first lesson was that the final output is not enough to judge. Validation had to move to every step along the way.

• Utterances - understanding what the customer actually said.
• Topic classification - confirming the inquiry was categorised correctly.
• Plans - checking that the plan the agent chose to follow is a valid one.
• Tools and actions - verifying the agent reached for the right tool to act on the customer’s behalf.
• Grounding - confirming it drew on the right information rather than plausible-looking alternatives.
• Response - concise, complete, accurate and coherent.

Limits of an LLM Judge

An LLM judge runs across those layers, and he was careful to bound what it can do. It is not a universal replacement for assertion.

Some checks still need deterministic assertions. Others need trace validation, retrieved metrics, and confirmation that the goal was genuinely completed rather than reported as complete.

That doubles the job. The test has to confirm the work was done and separately assess whether the agent held the right kind of conversation while doing it.

Phase 2: The Airline Lesson

Phase one validated agents Salesforce had built internally. Selling them to customers who then deployed them exposed a gap between what had been tested and what the agents were actually asked to do.

His example is the Singapore Airlines partnership across 2024 and 2025. The airline had industry terminology nobody had accounted for and customer inquiries that internal scenarios had not modelled.

The specific surprise is mundane and expensive. Customers writing to an airline rarely ask one question; they send several at once and expect the agent to work out what information it needs back from them.

Generalising from that, industries differ across customer flows, channels, terminology and personas, and each carries proprietary domain knowledge and its own knowledge articles.

The response was industry-specific test environments, with AI generating industry-specific customer flows and inquiries from real customer cases, then simulating how the agent behaves against them.

Phase 3: An Agent to Test an Agent

Phase three arrived with Agentforce Voice, a native voice solution where a customer calls a help centre and reaches an AI agent. That is where the testing model broke.

Callers arrive with different personas and backgrounds, some frustrated, many asking things the agent was never equipped to handle. Simulating that range needed an AI agent capable of playing the customer.

His example of the difficulty is a caller who wants to book an appointment and update their account details in the same conversation. One agent may not handle both, so it has to enlist another with different capabilities.

Generating a scenario like that programmatically is very hard, which is why his team partnered with the TestMu AI team, then LambdaTest, on an agent-to-agent testing framework. It produces the dynamic scenarios, exercises the handoff between agents, and evaluates whether the customer’s full request was satisfied.

Single Turn Is Not Enough

The structural point underneath phase three is that a turn is the wrong unit. A conversation can have a successful turn and then fail on the next inquiry from the same customer.

When that happens the journey failed, whatever the individual turns recorded. Completing one inquiry does not mean the job is done.

Test agents let his team follow intent as it develops, change what the test does mid-conversation, bring in an agent with different capabilities, and finally confirm the customer was satisfied with the outcome.

The Voice Variables

Voice adds a category of variables that never existed in a text channel, and his list is worth reading as a test plan.

• Accents and background noise - and whether the conversation transcribes correctly through both.
• Thinking on the spot - working out what the customer is asking and what information to bring.
• Handoff - routing to the agent that can actually help with the specific inquiry.
• Data retrieval - pulling from Salesforce to surface or complete what the customer asked for.
• Caller state - whether the customer is frustrated or patient, and whether they interrupted the agent mid-sentence.

Testing at Scale

Beyond whether the agent works, there is the question of whether it keeps working under pressure. Latency matters, and so does whether accuracy at high load matches accuracy at normal load.

Two more behaviours belong in the same tier. Whether the agent recovers when it hits an issue mid-conversation, and whether it escalates to a human correctly when it senses the inquiry is beyond it.

What is being assessed splits three ways: the quality of the agent, the quality of the system, and the experience of the human on the phone.

Phase 4: Packaging It Up

The current phase turns the internal lessons outward. The tooling went into Agentforce Studio so customers can tune and customise their own agents rather than filing tickets.

• Persona simulation - running different customer personas against their own agent.
• Custom evals - customer-defined evaluations of their own agent’s outcomes.
• CI/CD integration - the same evaluations running in their pipeline.
• Traces and run history - monitoring steps along the way and seeing performance over time, so a prompt-tuning change can be judged on whether accuracy moved.

The Lessons Learned

He closed the narrative with six lessons from two years of the journey.

• Deterministic tests still matter - they are not artifacts of an age that has passed, they need augmenting with the right tools, and evaluations plus agentic tests give you both.
• Validate steps and outcomes, not responses - confirm the agent completed what it claimed, did not divulge what it should not, and did not claim abilities it lacks.
• Make the environment representative - bake real-world scenarios into the testing architecture from the start, or discover the gap after deployment.
• Take industry language and grounded data seriously - the verbiage and customer-handling conventions differ enough between industries to change outcomes.
• Use AI to expand scenarios and monitor quality - manual testing across channels does not scale, and test agents took them to hundreds and thousands of concurrent calls.
• Put evaluations in CI/CD - agent accuracy fluctuates through the development lifecycle, so a final pre-release check is the wrong place for it.

The New Quality Loop

His summary is that agentic AI changed the unit under test. It is now a dynamic customer journey spanning tools, data, languages and operations rather than a feature with a boundary.

Dynamic tests are where the discipline is heading, and he repeated that this does not retire deterministic ones. The loop he described runs from realistic environments to scenarios generated from customer data and use cases, then evaluate, observe and enrich, feeding back into the next cycle.

Q & A Session

Three audience questions closed the session.

• What surprised you most, in the sense of something agents were unexpectedly bad at?

  John Liang: Retrieval among similar documents. Even grounded with the right data, an agent sometimes fails to produce the response you expect, and much of that traces back to how the knowledge is chunked. You can design an agent with the right prompt and feed it the right knowledge article and still watch it miss the answer. It surfaced in production with airline customers, whose feedback was that the agent could not find the right article among several similar ones. Agents locate plenty of related articles; picking the one a person would immediately judge most relevant is what they do neither accurately nor consistently.

• How do you see agent-to-agent testing working alongside Agentforce Studio for voice agents?

  John Liang: He drew the line at scale. Agentforce Studio is good for testing your own changes internally and is not yet something you can run at volume, which he named as something his team still needs to improve. The agent-to-agent framework covers the other half: accuracy testing at thousands of concurrent calls, monitoring whether the agent keeps fulfilling complex turn-by-turn requests, and detecting when quality has degraded or deviated from the bar.

• How do you measure agent accuracy at scale, at real customer volume?

  John Liang: Not by placing 5,000 real concurrent calls. The volume is simulated with virtual calls, and then roughly 50 to 60 of the actual calls made by those agents are taken for evaluation. Those are assessed across the dimensions the team cares about, meaning accuracy, conciseness, completeness and tone, along with the quality of the voice itself, with background noise simulated as part of the test.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.