Bringing Mobile App Validation in the Agentic Loop [Testμ 2026]

The Browser Blind Spot

The problem he opens with is not a missing feature. It is a mismatch between where the agents went and where the users are.

Hundreds of browser agents have shipped. The frontier models, the coding agents, the browser-use and computer-use tools all converge on driving a browser to complete an objective.

Meanwhile the hours an actual person spends are on a phone, inside apps. His framing of the mismatch is the line the session is built on.

The testing gap that leaves is not a rounding error at the edge of the market. It is the surface most of the revenue moves through.

Why Is Validation Still Manual?

The irony he points at is that agents already reach the simulator. Tools that build mobile apps, Lovable and Claude Code among them, launch one to check their own work.

So the build side of mobile is agentic. You can ask for an application and have it in fifteen minutes.

The validation side is not. You still write the test cases by hand, still write the XCUITest or the Appium code, and the testing process sits outside the loop entirely rather than inside it.

What Makes Mobile Harder?

Before the demo he answers why browser automation got solved first, in four parts.

• Authoring is slow - a single Appium or XCUITest flow takes real time to create before it can run in a regression pipeline, between locators, waits and platform-specific handling.
• Selectors do not hold - there is no DOM. You are working against an accessibility tree that shifts with the OS version, so a script that passed can stop being valid on the next run.
• Coverage explodes - cross-browser testing has a bounded matrix. Mobile has hundreds of OEMs producing hundreds of models, and bringing OS versions into the picture multiplies it again, with every build expected to work on all of them.
• Upkeep eats the budget - keeping existing cases green consumes the majority of the effort, which is what makes scaling coverage impossible rather than merely hard. You often cannot even determine what coverage you currently have.

Kane CLI on a Simulator

The first demo is deliberately plain. Kane CLI drives iOS simulators and Android emulators on your own machine, from the terminal.

The inputs are a build, an APK or a zip, and an objective. His was a checkout: place an order for a smartwatch on a shopping app.

It installed the application, analysed the objective and started acting, with its reasoning printed alongside the simulator. The steps that scrolled past were intent-based natural language rather than coordinates.

The distinction he stresses under that surface is that it is not guessing from pixels. Underneath the vision layer it is reading the application’s own layers to work out how the case should be automated deterministically on later runs.

He is candid about where demos usually break. Everything works on sign-in, and agents start failing once a flow gets nested or complex, which is why the hybrid handling matters more than the happy path.

One detail he flags against himself: because he supplied no delivery address, the agent invented one. In real use you pass that context in as variables and parameters, which is also what makes a run repeatable rather than a one-off success.

The objective was met in under three minutes, and exiting saved the artifacts, including the test MD that makes the run replayable.

Modelling Mobile Context

With the run finished he goes back over the four hard parts and says what the agent does with each.

• No DOM, a live hierarchy - the view hierarchy changes depending on where you are, so elements are resolved by meaning rather than by a fixed path.
• Hybrid surfaces - a single flow crossing native screens and WebViews is treated as one surface, with no switching between contexts and no working out which WebView is the live one.
• Device metrics - a case that passes on a large screen goes flaky on a small one purely because of scroll. Driving from the objective lets the agent work out what to scroll and when.
• Device context - dialogs, locale and network conditions become first-class steps rather than something patched in later.

Each step records its own context, which is the part that pays off on replay. When the app changes, the run fails loudly and precisely instead of failing somewhere vague.

One Objective Is Not a Loop

This is the turn in the session, and he makes it against his own demo. Meeting an objective is not yet agentic. He supplied the objective, the agent acted, and nothing closed.

What a loop needs is assurance that each test verifies an acceptance criterion, coverage that can be measured back to requirements, and a failure that traces to whichever criterion broke.

His formulation of the standard is worth borrowing regardless of tooling: a passing test should prove itself, and coverage should not be a count. Five hundred test cases is a number, not evidence of anything.

Requirement to Proof

Assurance is the name for that loop, and it runs from context to evidence: ingest a source, derive the use cases, design the tests, run them, report coverage, produce proof.

The stage most tools skip is the last one. Maintenance here means reconciling the suite as new context arrives, working out which existing tests should be archived and which edited, not only adding new ones on top.

The second demo runs that loop with Kane CLI installed as a skill inside Claude Code, working from a light PRD for an iOS shopping app covering authentication, catalog and cart, plus a set of out-of-scope items.

Asked in plain language to generate test cases from the PRD, Claude picked up the Kane CLI skill on its own and switched into the assurance flow for requirement-linked test design.

Why Not Feed the Codebase?

He pauses the demo to answer the objection he expects, and it is the most transferable argument in the session.

The instinct is to hand the agent your whole codebase so it can test the app thoroughly. His case against it is short: the code was written from a requirement, so testing that code with the code as context inherits the code’s bias.

You then cannot say whether a passing test is validating the requirement or merely agreeing with the implementation. The route he argues for goes requirement first, use cases derived from it, and tests designed from those.

The same reasoning explains why the flow derives use cases before it writes a single test, rather than handing the PRD to a model and asking for test cases directly.

18 Tests, 46 Criteria

The run produced use cases first: signing in to the shop, managing a shopping cart, browsing the product catalog, and others.

Then it stopped for a human. He reviewed them and approved three, leaving out the ones he judged too thin on PRD backing to be worth designing against, which is the human-in-the-loop checkpoint that keeps hallucinated use cases from propagating into a suite.

Design ran against only the PRD and those approved use cases, and produced 18 tests carrying 46 acceptance criteria, with gaps recorded and items flagged for attention.

Each test is a test MD: an ID, steps written as objective-based natural language intent, and the acceptance criteria each step verifies. Existing cases can be imported into the same structure rather than rewritten.

Authoring then happened live on the simulator, and four requirement-linked tests came back with something more useful than a green run. Two real failures surfaced: a password rule requiring at least eight characters, and a login attempt with invalid credentials. Both were reported as bugs found in the application itself.

The Evidence Pack

Alongside each authored test sits a result MD carrying the same natural-language steps plus a pass or fail proof, and behind that sits the evidence pack, served locally and opened in a viewer.

The property he emphasises is that it is readable by a person and by an agent, which is what lets the next run reason about the last one.

Opening a failing case showed the exact steps taken, the assertions made against authentication state, screenshots at each point, and an issues section describing what failed and why.

The coverage view is where the structure earns itself. A use case holds scenarios, each scenario holds acceptance criteria, and each criterion is marked verified by the specific test that proved it.

Where a criterion has not been proved, because a test was generated but never run, it reports as skipped rather than passing, and names the gap it leaves.

Designed vs Proven

The takeaway he closes on reframes coverage as two questions rather than one.

• Designed - how much of the requirement has tests designed against it. In his own demo that was three use cases out of seven, so four were never designed for at all.
• Proven - how many of those designed tests actually ran and passed on a real target.

Report one number and you hide the half that matters, which is the proven axis. It is a useful test to apply to any coverage figure you are handed, agentic or not.

His three closing points: mobile validation is in the agentic loop, context is modelled rather than left flaky, and coverage is now both designed and proven.

Q & A Session

The Q&A box filled up during the session, and five questions made it into the time remaining.

• What are the token economics of repeated agentic runs?

  Shantanu Wali: The problem is that an agentic run is not deterministic. Ask an agent to meet the same objective twice and it may reach it by a different path, consuming the same tokens again each time, which does not survive contact with a regression suite that runs continuously. Kane CLI authors the test once and then replays it without re-consuming tokens on the reasoning, spending them only on the vision calls needed to prove the run. Look at what has happened to pricing: frontier models launched cheap, prices have climbed over the past three months, and token economics is now a cost problem to handle rather than defer.

• Native apps have no DOM, so how does Kane CLI see the app, and how does it handle OTPs, permission pop-ups, biometrics and web views?

  Shantanu Wali: It combines what it sees on screen with methods that already exist in open-source mobile automation, which is also part of why the token cost stays down: the actions themselves are performed by established open-source methods rather than reasoned out from scratch. For hybrid apps it connects the Chrome DevTools protocol to interact with web views directly, so the usual work of detecting a new web view and switching context into it does not arise.

• Can Kane CLI test on simulators, emulators and real devices, and what are the best practices?

  Shantanu Wali: Real devices give you the real experience, because hardware is a genuine component of it. Simulators give quick feedback at lower cost. Validate on a simulator or emulator early in a launch, and bring real devices in at scale, validating on both. TestMu AI provides a cloud for emulators and simulators as well as real devices, so tests authored locally can execute there. On practices, my first recommendation is to add the Kane CLI skill to whichever coding agent you use and let it drive, because the skill already knows what needs doing. Building the pipeline yourself works too, through context injection and high-level objectives rather than step-by-step instructions. He treats this as a decision rather than a solved question.

• Can it automate a ride-hailing app that depends on GPS and sensor data?

  Shantanu Wali: Kane CLI can automate those cases with GPS mocking. To go back to the previous question, sensors and GPS are hardware components, so I strongly recommend running them on real devices to be confident the cases behave as intended. Complex scenarios are handled either way, whether the complexity is in the application or in the elements underneath it.

• We use Appium today. Can we leverage our existing test code?

  Shantanu Wali: Yes, by passing the existing cases to Kane CLI as context through your coding agent, so you are not rediscovering cases you already have. The caveat is the same argument I made against feeding in a codebase: I do not recommend it as the best practice, because the point of the loop is the assurance layer rather than the authored tests. Existing test cases are better context than code, though, so they can seed the use cases, acceptance criteria and scenarios the design pass needs. He is straight about that caveat.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.