Agentic Testing Life Cycle: The QE Loop for AI Agents

What Is the Agent Life Cycle?

The agent life cycle is the sequence an autonomous agent moves through in production. It takes an open-ended instruction, plans an approach, calls tools, writes files or hits APIs, and then reports what it did.

Stated that way it sounds like any other request path. Three properties separate it from one, and each removes something a traditional test relied on.

• The interface is open - the input is natural language and the output is whatever the agent decided to produce, which might be prose, a file, a ticket, an API call or a refund. There is no argument list to assert against.
• The behaviour is non-deterministic - the same prompt twice can take two different paths, so a test that pins the path fails constantly while a test that pins nothing proves nothing.
• The agent acts on the world - it is not returning a value for you to inspect. It is doing things, some of which cost money and some of which cannot be undone.

The practical consequence is that most teams shipping agents grade them by reading what the agent said about itself, in a transcript or in a judge score over the final message. That is the one account with a reason to be wrong.

The failure it hides has a shape. An agent reports that it issued the refund and notified the customer; it issued the refund and notified nobody, and nothing in the transcript reveals the difference. How that pressure reshapes both delivery and testing is the subject of agentic SDLC vs STLC.

What Is the Agentic Testing Life Cycle?

The agentic testing life cycle is the quality engineering process rebuilt around evidence instead of assertions. Rather than writing tests against a specification, the loop derives them from the agent, runs them against the live agent, and grades each claim on what it observed while the agent worked.

The classic testing life cycle survives in outline. What changes is the unit of judgement, which stops being a test case with an assertion behind it and becomes a criterion, an individually gradable claim with evidence attached.

That shift is what lets the loop say something a pass rate cannot. A run reports what was expected, what happened, a quote as evidence, and separately what it could not check at all.

TestMu AI ships this loop as Agent Assurance, which covers both the agents that act and the agents that talk. The conversational half, covering chat, voice and phone agents, is generally available as agent testing; the autonomous half runs from your terminal as rook and is pre-alpha at the time of writing.

Note: Grade an agent on what it did rather than what it said, with per-criterion evidence and an explicit count of what could not be checked. Get early access to TestMu AI Agent Assurance!

How Does the QE Loop Work?

Six phases carry the loop. In the terminal they are commands and in CI they are subcommands, and they do not have to be run in order, because asking for a later step plans the earlier ones first and shows the cost before anything is spent.

• Discover - reads the codebase and works out which agents live there and what they do, along two deliberately separate paths described in the next section.
• Generate - writes scenarios in three families: functional for the things the agent is for, non-functional for latency, cost and robustness under malformed input, and adversarial for prompt injection, instruction override, tool misuse and policy boundary probes.
• Profile - captures the one thing the loop cannot derive, which is how to invoke your agent. You paste a request or a command, it tells you what it understood field by field, and it invokes once with a trivial goal to prove the profile works before a suite is spent.
• Run - executes scenarios against the live agent several at a time, watching the filesystem under the declared paths, the artifacts produced, and the tool calls made.
• Judge - a judging subagent grades each criterion by investigating rather than accepting, reading files and inspecting artifacts to establish what actually happened.
• Report - per-criterion verdicts with quoted evidence, the assurance gap, and what changed since the last run.

Multiple agents do the work at different stages. A discovery subagent reads code the deterministic pass cannot interpret, and a fleet of specialised evaluators grades the agent under test at machine speed, which is what makes a generated suite practical to run on every change.

Cost is visible before it is incurred rather than after. Usage is metered in credits, a run states its estimate up front in the shape of a scenario count and a credit figure, and a budget command caps what a session may spend.

A profile is also how you compare two configurations of the same agent, since an agent can carry several, such as staging, production and a faster model. The continuous version of this loop, running past the merge gate and into production, is covered in continuous AI agent testing.

What Context Does the QE Process Read?

Discovery runs along two paths that are kept separate on purpose, because mixing them is how a harness starts inventing.

• Deterministic discovery - agent manifests, framework conventions, agent definition files, MCP server configurations and tool tables, with no model involved. For an MCP server it connects and asks what tools the server really has rather than trusting what the config claims.
• Model-assisted exploration - for a plain codebase with a prompt sitting in a string and a tool table nearby, a discovery subagent reads the code to work out the shape of the agent.

The governing rule is that discovery never invents. Where an agent declares nothing about a tool, the field comes back as unknown rather than as a plausible guess, because a fabricated tool surface produces fabricated verdicts downstream.

Code is only one of the three contexts a QE process for agents has to hold, and the other two are what most eval harnesses miss.

• Business intent - what the agent is supposed to be for, expressed as expected behaviour with validation rules and edge cases. Coverage measured against declared behaviour is a statement about the agent’s purpose; coverage measured against a pile of scenarios is a statement about volume.
• The change under review - an agent that reads pull requests reads whatever is in them, which makes an instruction injected into a commit body a real attack path rather than a hypothetical. Adversarial scenarios generated as a first-class family are the reason that path gets tested at all.
• The agent’s own record - tool calls, artifacts and filesystem effects, treated as evidence to check against rather than as a report to accept.

Some of this context becomes durable rather than per-run. In the Agent Assurance cloud experience an agent’s derived context is versioned with a diff between versions, so a team can answer what changed about the agent rather than only what changed about the code. That surface is designed and not yet shipped, so treat it as intended product.

What Does a Verdict Actually Prove?

A criterion resolves to one of three outcomes rather than the usual two, and the third is the one that makes the other two mean something.

Excluding the third from the denominator is what keeps a pass rate comparable across runs, teams and agents. Every number in the report is then a number somebody looked at, and a verdict can legitimately read as a pass alongside a statement of what nobody checked.

The percentage of criteria a run could not verify is the assurance gap, reported beside the pass rate. It turns an invisible problem into an arguable number, because a team that knows how much of its agents’ behaviour it can actually prove has something it can improve on purpose.

Measured on the three reference agents that ship with the product, on suites of the same shape, the gap tracks how much each agent records about itself.

Those figures come from TestMu AI’s own reference agents, built to demonstrate the mechanism rather than to benchmark anything, so read them as an illustration of the pattern instead of as a comparison. The pattern itself is the argument: the more an agent writes down about what it did, the more of its behaviour can be proved.

That makes the gap actionable by the team that owns the agent. Adding an audit log of tool calls is an observability change in the agent, and the number moves because the agent became more verifiable.

How Do You Gate a Build on an Agent Run?

Headless subcommands and meaningful exit codes carry the CI half of the life cycle. The distinction that matters is between a finding about your agent and a failure of the harness to look.

A build should stop on either 1 or 2, and only 2 is a statement about your agent. Anything unverifiable does not fail the build at all, because making a harness limitation look like a product bug is how a team learns to ignore its own gate.

Resist gating on the assurance gap on day one. Measure it for a few weeks, watch it fall as the agent records more about itself, and then set a threshold the team has earned rather than one picked from nothing.

Point it at staging. The agent under test is yours and its writes are real, so a suite that invokes it the way a user would cannot roll anything back. Agent Assurance states how many write tools an agent declares and asks for approval once before a run, grants are per target rather than global, and judges are instructed to verify without changing anything.

The wider practice of standing this up as a team discipline, rather than as one pipeline step, is covered in our guide to agentic quality assurance.

Note: Run one generated suite against a staging agent and read the assurance gap before you tune anything else. Read the Agent Testing docs

Conclusion

Point a generated suite at one agent you already ship, in staging, and read the assurance gap before you change anything. A high first number is the expected result, and it tells you how much of your current confidence was inference.

Then make that agent record its tool calls and run the same suite again. The conversational half of Agent Assurance is generally available today, and the launch story for the whole product is in introducing Agent Assurance.

Agent Assurance runs from your terminal as rook, a pre-alpha tool whose source is public at github.com/LambdaTest/rook.