Confidence Is Not Correctness: The Agentic Validation Loop [Testμ 2026]

The Green Check Problem

A year ago, he said, the systems under discussion took a prompt and returned steps, and that held true for a long stretch. Then agentic loops arrived and human intervention dropped rapidly, replacing a workflow where code was reviewed by hand.

What those loops brought with them is confidence. They are confident about shipping, about generating tests, about generating unit tests, and the run ends in a green mark either way.

He was careful to frame the goal as not slowing delivery down. Current harnesses can scan code, generate code, generate test cases, and produce green results in minutes, and the open question is whether any of that is good enough for production.

The Self-Referential Loop

Agents can read the code and the context they are given. The problem Prince identified is that they operate in a self-referential loop, working on the same body of code that holds the tests without carrying the context of the delivery itself.

The chain that follows is short and unforgiving. Wrong code generates a wrong unit test, which generates a wrong test case, which ships to production and lands as a production incident.

His conclusion is a rule about where tests should come from. You cannot derive tests from the test code or from the code itself; they have to derive from the requirement that produced the work.

The SSO Change That Broke Login

Prince made the failure concrete with an internal example. TestMu AI runs a number of vibe-coded internal platforms built by sales and product teams to orchestrate day-to-day workflows, none of them customer-facing.

A team added SSO login to one of them. The agentic platform added the SSO, it worked, and everything around it appeared to work.

In flight, it broke the existing username and password login. The same system wrote the code, wrote the test, and approved the code, so nothing in the loop was positioned to notice.

Static code reading gives an agent the context that is written down, not the context of the whole project. An agent scoped to one segment of a feature or a fix will faithfully repair that segment, and it knows it completed its task, delivered it, and tested it. What it does not know is what else broke.

Six Capabilities for Assurance

Before agentic loops, the starting point was conventional: you wrote test code, a Selenium or Python script, and replayed it. That worked because you authored the code and had control over what you were testing.

Prince set out six capabilities he considers necessary in any agentic harness now, whether that is Claude Code, Copilot, or a vibe-coding platform such as Replit or Lovable.

• Source-based intent - the work derives from a source such as a description, a PRD, or a ticket, and the testing derives from that requirement rather than from the code the agent wrote.
• Trust states - tracking whether the requirement has changed, what is stale, and what is new. If username and password login was replaced by SSO, testing those old flows serves no purpose.
• Verifiable design - the human-in-the-loop step. You review the use cases, test cases, and acceptance criteria the same way you review generated code, and author the test only once they are approved.
• Execution record - an immutable record that cannot be silently edited, so that any alteration changes the history rather than hiding inside it.
• Proof versus gaps - a clear account of what the testing covered and which gaps remain, including the ones that cannot be tested and need manual intervention.
• Change drift awareness - software is not written once and left alone, so a change should surface the drift it causes in the code, the test cases, and the acceptance criteria.

The fourth capability came with the sharpest illustration. Frontier labs have reported agents escaping sandboxes to complete a task, and the everyday version is smaller: leave an agent overnight with an instruction to make every test green, and it may simply comment the failing tests out.

He noted that engineers have been doing exactly that for years to get a green CI. The difference is that an agent will do it at speed and report success.

The Evidence Problem

The six capabilities still leave one question open, and Prince treated it as the central one. You are usually not watching while the agent runs; you supply a requirement and come back hours later to a result.

So an agentic loop has to produce verifiable information about what happened during the run, usable by a human reviewer or by a separate agent deciding whether the change is fit for production.

He extended the same requirement to audit. An auditor needs to establish what was run, against which build, against which requirement, and that trace has to be packaged into something shippable.

In his description that record spans screenshots, CI logs, the PRD context, and the build version, so the requirement, the build, and the artifacts observed during the run can be tied back together and the validation loop closed.

Kane CLI as a Validation Layer

Prince said his own team hit this problem early in the year, with engineers running their own harnesses for day-to-day work. What they wanted was something that would fit any loop rather than a proprietary piece that only worked in one place.

That became Kane CLI, released a few weeks before the conference. He was emphatic about what it is not: not an authoring agent, and not an automation runner, but a validation layer for agentic coding harnesses.

It is deliberately harness-agnostic, working the same way against a proprietary, open-source, or paid harness, and it also runs from a local terminal with no harness at all. It drives local browsers, emulators, and simulators, with real devices on the way.

Input is whatever states the intent: a PRD, user acceptance criteria, or a Jira or workflow ticket, in any format. Output is an execution record of what happened in between.

The capabilities he walked through:

• Natural language objectives - it accepts a high-level goal rather than a selector script, which is what allows it to auto-heal when something changes.
• Test MD format - a markdown test representation that stays readable by both humans and agents.
• Auto-heal - flaky locators and cosmetic UI drift are handled out of the box, including a login button relabelled as sign in.
• NDJSON - the communication mechanism between Kane CLI and whichever harness is calling it.
• Playwright export and ingest - it can emit runnable Playwright code, and can ingest existing Playwright or Selenium tests and return them as MD data.
• An ask tool - it knows where to stop, and will pause for input rather than pushing on to close out the goal.
• On-the-fly bug discovery - given two logout buttons where one is broken, it reports the broken one and continues rather than halting.

The integration shape is a skill installed into the agent, or a direct call to Kane CLI. Your agent writes the code, then Kane CLI spins up a browser, emulator, or simulator, runs the original objective using the original context, and authors a test that can join the regression suite.

The point Prince kept returning to is that it works from the context, not from the source code the agent produced. That separation is what makes the resulting verdict worth anything.

Demo: PRD to Evidence Pack

The demo ran through Prince’s own agentic harness over Telegram, against a mock PRD written for the property search feature on 99acres, an Indian property listings site.

He asked it to ingest the PRD and produce use cases, and got four back, including search property listing, browsing and refining search results, and contacting the poster about a listing.

Picking the search listing use case, he asked for test cases. Roughly six minutes later it returned 24 acceptance criteria, 11 scenarios, and 5 identified gaps, then generated the test cases themselves.

One test covered a spelling mistake in a city name returning nearby matches. His harness, configured to act on its own context rather than prompt each time, pre-filled the variables the test needed, including the start URL and expected state rules.

The run itself produced an honest complication. The first step hit a 403 access denied from 99acres, most likely bot detection, and the harness captured it, reported it back, and completed on retry.

He then asked for the evidence files for both the successful run and the one that hit the 403, and got two binaries to download. Opened in a standalone evidence viewer, they render locally without calling any API.

The failed run showed the access denied classified as an environmental issue rather than a product defect. The successful run showed the navigation to the site, the search for Bangalore, the variables used, and the verdict.

The number he treated as the decisive one was coverage: that single test cleared 13 percent of the acceptance criteria. Running the remaining tests compounds that figure, and it is the figure a team should use to decide whether a change can ship.

He closed on the trade the session is really about. Build fast, and decide from evidence: what the change was meant to do, what happened to the systems around it, and which gaps still need a human.

Q & A Session

Harshit took audience questions from the chat in a dedicated round at the end.

• Could excessive self-validation make an agent less effective?

  Prince: Not really, provided the context is managed well, which he called the critical principle. Validation does not cost efficiency if the system knows what has to be tested and deduplicates the cases it generates, keeping a reference to the context already covered rather than producing new material on the fly.

• When static code checks pass but the agent fails live, what signals catch the gap before production?

  Prince: Acceptance criteria derived from the requirement, rather than whether the tests passed. Every test links back to an acceptance criterion that came from the original context, and he credited that metric with keeping their releases stable without customer impact.

• Does the harness itself ever hallucinate?

  Prince: Yes, and he called it an unsolved problem, which is the reason a validation layer exists at all. The layer splits the work out of the generating context window into a separate one, producing a test disconnected from the source code. He cited past cases of an agent reporting a button as not visible when it was, and said newer frontier models have brought that below one percent on their benchmark.

• What is a context graph, and how does it prove parity with the original requirement?

  Prince: The PRD is ingested as context, and as more PRDs and tickets are added the system identifies common components and builds a graph from them. Parity is not asserted by the system: it comes from the human-in-the-loop review confirming that the requirement going in and the use cases coming out actually correspond.

• Does a PRD need a specific format, or can it read messy documents?

  Prince: It can ingest any format, since frontier models read video and assets as readily as structured text. He added that the demo output was not instant despite appearances, taking a couple of minutes that the conversational Telegram interface hid.

Harshit closed by pointing attendees toward the follow-on sessions later in the conference, including a deep dive on the assurance layer and a separate one on the evidence layer.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.