Redefining Quality Leadership in an Agentic World [Testμ 2026]

The 94.2 Percent That Cost $400,000

The system was a multi-agent transactional workspace integration designed to orchestrate customer refunds, the kind of flow anyone using a payment gateway depends on when a transaction fails.

The legacy test reports read as a success. A pass rate of 94.2 percent would normally be taken as excellent, with the major test cases green and nothing obviously wrong.

Then the agent entered a recursive prompt loop at midnight and repeated transactions ran unchecked. The bill reached $400,000 in refunds before alerts triggered.

Her diagnosis was structural rather than a one-off bug. This is what happens when a deterministic framework is applied to a probabilistic environment, where pass and fail no longer describe the thing you need to know.

The questions that replace it are about degree. How accurate is the result, is the customer served by it, and does the system return a consistent answer or a different one each run.

Breaking a Probabilistic System

Testers know how to break deterministic software. Negative paths, dirty testing, monkey testing, and a general instinct for finding the edge that makes something snap.

Sobhitha put the harder question directly: how do you challenge a probabilistic system, where the same prompt produces a different response each time?

She grounded it in the chat and voice agents her team works on at Salesforce, where responses vary not only per prompt but per region. A customer in one market may expect something casual, another market may expect formality and a direct answer.

That opens questions a pass or fail cannot reach. How is customer feedback weighed, how does the agent decide between escalating and resolving, and how do you establish whether the issue was genuinely fixed.

Her sharpest framing of the problem is that an AI agent can close a case easily enough. Whether it solved the customer’s problem is a separate question, and nothing in a green report distinguishes the two.

The Trust Equation

The first of her three parts is a framework for treating trust as something measurable rather than a feeling. It rests on three components.

• Alignment - the degree of adherence to semantic criteria, assessed with evals where one model scores another model’s responses. She was careful to separate meeting your own accuracy threshold from meeting the customer’s.
• Predictability - whether the system returns consistent results across many prompt iterations at scale, which is what makes it reliable rather than merely capable.
• Blast radius - the familiar question of what a change touches and which regression areas it puts at risk.

The arrangement carries the argument. Alignment and predictability belong in the numerator and should be driven as high as possible, while blast radius belongs in the denominator and should be shrunk within defined boundaries.

Expressed that way, trust becomes a hard metric that can be measured consistently across teams rather than asserted at a release meeting.

Core Trust Indicators

Underneath the equation sit four indicators she treats as the practical measurements.

• Conversational accuracy - whether the agent judges correctly when it cannot solve a problem and should transfer to a human, and when a case is genuinely resolved and can be closed.
• Intent continuity - whether the response matched what the customer actually wanted. An agent can be technically and grammatically correct while missing a frustrated customer entirely, which is why empathy has to be trained rather than assumed.
• Variance - system divergence across multiple runs, contained within a range rather than swinging between outputs. She cited a variance around 0.04 against accuracy close to 99 percent as the shape to aim for.
• Mitigation - whether edge cases are contained and fixes stay clear of surrounding APIs, through gated isolation and feature flags that can be closed quickly when a customer hits a problem.

Her point about this list is that most of it was not previously a quality engineer’s concern. These are now the primary indicators, and the measures they replace no longer carry the same weight.

The Agentic Risk Matrix

Sobhitha organises risk by agency level, because what can go wrong changes as an agent is given more latitude.

The rule she draws across the three is that rising agency requires stronger isolation and stricter boundaries. Without them, the predictability of the outcome degrades in proportion to the freedom granted.

Aligning Upward and Downward

The second part of the talk is about what changes for leaders, and she splits it by direction.

Aligning upward means handling executive expectations, where the asks are increased productivity, faster go-to-market, and more pull requests per day. The corporate expectation is effectively zero errors.

The leadership job is translating that into practical probability ranges, since absolute zero is not always reachable once technical outcomes meet real customer expectations.

Aligning downward is about what the team is measured on. Counting scripts or pull requests is the wrong instrument, and she named what to track instead:

• Feedback loop length - whether automation runs faster and returns results more frequently than it used to.
• Self-healing - whether it is actually in place rather than aspirational.
• Consistency across a sprint - whether running daily through a two-week sprint produces stable results.
• Defect leakage ratio - how much reaches production, rather than how much was written.

Between those two directions sits what she called the reality gap. She cited research putting the gap between executive expectations and actual agentic performance at 78 percent.

Closing it means translating software reliability into commercial terms, and adopting impact-based regression in place of exhaustive suites. Understand the blast radius, prioritise the tests the change actually touches, and run those.

Chaos Audits

Her team replaced classical ticketing with something she calls chaos audits, aimed at the burnout that follows from chasing productivity against unpredictable systems.

Engineers are rewarded for inducing specific agent exceptions. An agent that threw many exceptions in one sprint and measurably fewer by the next is the outcome the incentive points at, alongside building fallback states and fast rollback paths.

The results she reported were a 98 percent reduction in production loops and a 65 percent drop in burnout.

She framed the mechanism as psychological safety. The team gets an environment where failing fast is the expected behaviour, followed by course correction into the next release.

Resource allocation shifts to match. Manual testing falls to roughly a fifth of the effort, automation takes around a third, and statistical evaluation plus guardrail modelling claim the share that manual work gives up.

The QE Career Pathway

The third part answers what happens to the quality engineer, laid out as four stages.

• Automator - writing deterministic scripts against tools such as Selenium, Cypress and Playwright.
• Evaluator - building semantic checks and tracking variance and predictability statistics across test sets at scale. She placed most quality engineers here today.
• Architect - configuring multi-agent models, implementing safeguards, rollback and self-healing, and deciding which tasks can be autonomous and which keep a human review.
• Guardian - setting operational context boundaries, ethical baselines and compliance systems across jurisdictions with different AI and privacy law.

The architect stage is not the traditional QE or development architect role. She used the term trust architect, and the distinguishing work is judging where autonomy is appropriate rather than building the automation itself.

Her team’s reskilling programmes, run through mandatory training and certification, produced a 68 percent shift from legacy manual work toward AI-enabled evaluation and a 42 percent productivity gain.

Three Pillars

Supporting that pathway are three areas she recommends investing in now.

• Semantic competence - tracking logical flows, prompts and context state consistency instead of reading a pass or fail.
• Statistical rigour - working in scores across dynamic evaluation matrices that shift by region and domain, since e-commerce weights speed while finance weights accuracy.
• Compliance and ethics - building systems with low bias that hold to regional law, avoid unnecessary data retention, and do not trade away security.

Her conclusion was that the industry does not need engineers who passively write or run scripts. It needs people who architect trust systems, and who earn a say in system design rather than receiving it at the end.

Q & A Session

Nikhil put audience questions to Sobhitha in a dedicated round at the end.

• How do leadership and team structures change as agents reshape the workforce?

  Sobhitha: The questions worth answering are whether you are automating the right things, whether regression suites are genuinely autonomous, whether the feedback loop is short enough to ship and answer customers quickly, and whether self-healing is in place. The traditional mode of quality engineering no longer works, and the shift in mindset matters because the goal is not survival but thriving.

• When agents test agents, what does a quality leader still own?

  Sobhitha: Agents handle the majority of testing, but the agency levels decide where humans stay. Manual testing shrinks without disappearing, because enterprise product suites carry workflows cutting across product lines and no agent holds all that domain context. Agents can run straightforward regression and functional validation autonomously; complex customer workflows still need people, and human involvement shifts toward requirements and design.

• How does blast radius affect trust, and how do you minimise it?

  Sobhitha: It depends on the change, and it rises with the number of integration points and the complexity of the workflows. A single change can reach customers across regions who differ by culture, demography and age. Keeping it low comes down to gates and feature flags: how quickly you can roll back, and whether a feature can be masked so only the customers who want it receive it.

• Which metrics actually belong on a quality leader’s dashboard now?

  Sobhitha: Consistency, predictability, accuracy and variance, plus whether token usage is justified by the effective output compared with the pre-AI era. On coverage she was blunt that total test coverage no longer carries the weight it did. What matters is targeted coverage driven by change impact and blast radius, alongside a shorter feedback loop and faster automation run times.

She also noted that AI is blurring the lines between product, engineering, quality and support roles, pointing to forward deployed engineers as a role that spans all four. Prompting skill, in her reading, now opens doors that once required deep coding knowledge.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.