Agentic AI Changed What We Test, And How We Test It [Testμ 2026]

The Train and the Driver

Pooja Oza frames traditional testing as a train on a fixed track running from point A to point B. Flip a switch or shift the track by an inch and the entire run derails and fails.

That train is exact-match automation: static scripts checking static DOM elements.

An agentic system behaves like an autonomous driver instead. Given the same destination it changes routes dynamically, stops for pedestrians and reroutes around traffic jams, so no two drives look identical while the goal is still reached safely.

Her conclusion from the analogy is direct: because the software now behaves like the autonomous driver, quality engineering can no longer rely on fixed-track scripts.

The session’s aim follows from that, moving from brittle exact-match checks to evaluating dynamic trajectories against strict safety policies and behavioural bounds.

The Shift in What We Test

Her core framing is that what we test changed before how we tested changed.

Traditional testing asks one question, does the output match, resting on an exact-match oracle where a specific input yields the same output every time along a single deterministic path.

With agentic systems, non-determinism means the execution path can vary widely across runs while still achieving the same goal, which breaks that single question.

The question shifts to whether the agent’s behaviour stayed within acceptable bounds.

The talk’s roadmap follows: break the exact-match assumption, measure behaviour across an envelope, and build real release confidence.

The Bounded Behavior Envelope

Instead of checking identical step execution, she validates the agent inside a bounded behaviour envelope defined by three strict gates.

Gate one asks whether the agent achieved the user goal. Gate two asks whether it stayed within authorization, business and safety limits. Gate three asks whether every decision and generated answer is grounded in verifiable fact.

Her summary of the model is the sentence worth keeping: the path may vary from run to run, but the contract stays fixed.

The oracle checks that the behavioural contract holds, which means user intent preserved, safety policies adhered to, and an explainable execution trace maintained.

Three Valid Paths, One Goal

The worked example is a user saying “move my 3:00 p.m. appointment to tomorrow morning”. Depending on the environment, three different agent actions are all valid.

In traditional testing those three traces would look like inconsistent failures. In agentic testing all three pass, because intent is preserved, policy is obeyed and the trace is fully explainable.

Five Evaluation Dimensions

Fluency is not correctness. An answer can look polished, articulate and completely plausible while failing safety, policy or grounding constraints.

Binary pass and fail therefore gives way to a multi-dimensional spectrum, evaluated across five dimensions: outcome, meaning whether the agent resolved the user goal; grounding, meaning whether the response is strictly tied to verified authoritative data rather than hallucinated facts; the allowed action and tool-call sequence; policy compliance covering safety, authorization limits and privacy bounds; and reliability, meaning performant and resilient under latency or transient errors.

She puts a question to the audience worth carrying back to your own architecture: if an agent produces a wrong but highly polished response, which control catches it first?

Relying on user bug reports means the failure has already reached production. Structured pre-release testing plus real-time trace telemetry is how subtle high-risk failures get caught before customers see them.

The Ten-Second Quiz

The quiz put to the audience: a customer asks whether they can return a discounted item after 60 days, and two replies are shown.

Reply A is direct and confident, granting a 90-day return window on discounted items, and it is entirely fabricated.

Reply B qualifies the policy boundaries and asks for context: final sale is excluded, other online items are 30 days, and was this a final sale?

The lesson is that fluency is not evidence. An answer can be articulate, grammatically flawless and sound authoritative while being completely wrong, so oracles must explicitly validate source grounding and policy constraints.

She cites real-world failures with a common root cause of missing behavioural controls, including contradictory news summaries in early iOS 18.3 builds and hallucinated advice in public search and city chatbots. Both are second-hand press incidents recalled without citation.

When systems lack source fidelity, risk states or authoritative grounding, confident outputs become major quality regressions, which is her argument for explicit guardrails and domain-specific regression tests that block ungrounded behaviour.

The Blank Page Problem

Traditionally an engineer starts with a blank page and writes every test case by hand. It is slow, and coverage is limited to what one person can remember.

With AI, teams do not start from scratch. It instantly suggests execution paths, edge cases and retry scenarios based on product requirements.

The engineer’s role does not disappear, it moves up a level: verifying the risk, checking the facts and making sure assertions are solid before anything ships.

False Greens in Self-Healing

Self-healing locators solve UI fragility, and blind healing introduces the failure mode she calls the false green.

Her hypothetical: a checkout page updates its layout, the script cannot find “place order”, so it swaps the locator to another nearby clickable element, “save for later”. The test clicks it and passes green, and no purchase was ever made.

Her reframing of the question is the useful part: not whether the tool can find another button to click, but whether the actual business action happened.

In governed healing the tool may propose the new locator but must prove the outcome. It verifies the database state, confirms the order receipt was generated, and records the repair in the trace for engineering review.

A test run should not turn green because a selector was fixed. It turns green when the user outcome is proven.

Confidence Boosters to Fund

AI gives speed, through drafting tests, clustering failures and spotting patterns across traces. Speed is an input, not release approval.

Three places stay irreplaceably human. People define harm by setting safety boundaries and risk tolerances, people own accountability by authorizing high-impact and irreversible actions, and people sign off on releases with the final go or no-go call.

Four confidence boosters go on the slide, with the audience asked which they would fund first: golden data sets, traces, adversarial testing and human review.

Her funding rule is to attack the team’s largest uncertainty. Repeatability problems call for golden data sets, unknown behaviour calls for trace telemetry, security exposure calls for adversarial red teaming, and irreversible business impact calls for governed human review.

The Media Player Blueprint

She presents this section explicitly as an architectural blueprint and conceptual reference model, showing how the principles apply to a complex domain rather than describing a shipped system.

The streaming features that break traditional automation are timeline markers, auto-skipping recaps, intro cues, dynamic ad insertions and credit rolls. A standard script just checks a button exists and clicks it, but under network throttling or a stalled video the button may vanish, or the click happens without the play head ever moving.

For recap markers, the agent triggers a skip as the show loads, and the test confirms the video jumps to the exact right second with clean audio and zero lag.

For intro cues, even when the internet slows and video quality fluctuates, the skip-intro prompt must appear on time, and clicking it must land cleanly at the start of the episode with synced sound and no frozen frames.

For ad breaks, the system checks behind the scenes that ad tracking fires correctly and that the player switches between episode and ad smoothly without crashing, freezing or glitching.

For credits, the video shrinks to picture-in-picture while silently preloading the next episode so it starts instantly without buffering. The framework reviews the entire playback log, marks the test pass if everything worked, and flags the exact timestamps for engineering triage when anything lagged or skipped off target.

Four Proof Points to Ship

In the traditional model the pipeline runs a suite, looks for 100% green assertion checks and automatically triggers deployment. Fast, and completely blind to behavioural drift.

Here is what that model misses. The script only checks that the skip-intro button was clicked, and it finishes green while the stream dropped frames, lost audio-video sync, or failed to trigger its ad telemetry beacon.

Her first proof point is analysing the full playback trajectory, verifying that presentation timestamps landed within the expected time frame and that codecs switched cleanly without a crash.

The second is confirming the agent stayed strictly within policy, for example never skipping mandatory mid-roll ad pods or exceeding buffer limits.

The third is verifying that telemetry beacons for start, midpoint and complete fired and logged verifiable proof in the observability store.

The fourth applies to high-impact changes or detected regressions: a human reviews the packaged trace bundle before authorizing the release.

Three Questions Per PR

For engineers, tests stop caring how the agent reached the outcome and start checking whether the outcome held, which cuts the rewrite cycle. No more chasing a broken selector on every UI shift, or rewriting the same test two or three times a sprint because the click path changed rather than the behaviour.

For QA and SDET roles, AI proposing coverage removes the blank page problem rather than the judgement. Time moves to reviewing, verifying and owning the evidence trail, and what comes back is a defensible record behind every release instead of a green check mark that may not mean what everyone assumes.

For leadership this is a “why did we ship” story, and when something goes wrong the answer needs to be a trace rather than a shrug. What it buys is fewer incidents, faster release cycles and a paper trail.

On cost she is deliberately unquantified. Prevention is cheaper than an incident, and she refuses to put a dollar figure on it because she has no validated number to back one up.

Her three questions for the very next pull request: if an AI test passes, did you actually check the logs, meaning network calls, database state and timestamps? Did you set hard guardrails so the agent never exceeds token budgets, never retries an API call in an infinite loop, and escalates immediately on an unknown state?

The third is the closing habit. Use AI for the heavy lifting, letting it generate 20 edge cases and cluster failed logs in seconds, and keep the keys, because the human engineer is the only one who decides whether the risk profile is safe enough to ship to real customers.

Q & A Session

Five audience questions followed, and Pooja Oza scoped every answer to her own part of the organisation.

• Do you run red-teaming pipelines?

  Pooja Oza: Not that I know of, and my side of the org does not have red teaming yet. She picks this one up from the Q&A box herself before the moderator works through the queue, scopes it explicitly to her own part of the organisation and describes no plans to add it. Worth noting, because adversarial red teaming is one of the four confidence boosters she had just recommended funding when security exposure is a team’s biggest uncertainty.

• How do you test and cap the token consumption of an agentic testing loop to prevent excess costs during deep exploratory cycles?

  Pooja Oza: We are still in an exploratory phase, still getting the agentic automation ready and still trying to establish ROI, so I have no number to give you. What we do track is how regression cycles consume tokens, and more urgently whether the agent has entered a continuous loop on an API. Proactively screening telemetry for infinite loops and bottlenecks is how we avoid the scenarios you regret later. I expect a clearer picture at the end of the current cycle, and the work covers internal operations apps as well as consumer-facing ones.

• When a single user request can trigger multiple AI calls, microservices, databases and external APIs, should we measure the cost of a business outcome rather than the cost of individual cloud resources?

  Pooja Oza: When my teams look at frameworks we segregate groups of tests, API against databases against microservices, so cost and ROI can be gauged per segment across teams. Everyone eventually has to present to their executives how the work affects the business and improves ROI, so the measurement has to map onto business outcomes. She agrees with the premise of the question.

• What skills gap hit your team hardest in this shift?

  Pooja Oza: Mindset before tooling. My team hears that quality engineers will no longer be needed, so I constantly remind them that they remain the authoritative people for releases and go/no-go decisions, while also insisting they have to use AI because that is where the future is. The second gap is tool churn: multiple tools arriving, something new almost every week and new architectures appearing constantly, so as soon as you start a proof of concept on one, something else has already landed.

• Where do humans still clearly beat agents in your testing today?

  Pooja Oza: On my internal projects agents are not used and human intervention is still heavy. Integration testing and the customer user journeys running across the applications I test carry too much risk to let an agent make the release decision. Migration work is another place where an agentic architecture would not suffice, or would not return as much ROI as working with a human team. Humans remain the gatekeepers guarding quality and making the go/no-go call.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.