What Does an Agent-Run Test Suite Actually Cost Compared to Plain CI

The Two Cost Models

A scripted suite is expensive to write and nearly free to run. An agent-run suite is nearly free to write and costs something every single time it executes. Everything else follows from that inversion.

• Scripted, up front - an engineer writes the spec, the selectors, the fixtures, and the waits, and that time is spent before the first run.
• Scripted, ongoing - the suite is cheap per execution and expensive per product change, because the product changing is what breaks it.
• Agent-run, up front - a sentence describing what should happen, which is minutes rather than days.
• Agent-run, ongoing - inference on every execution, so the cost scales with how often you run it rather than with how often the product changes.

Teams comparing the two almost always compare the ongoing columns and ignore the up-front ones. That is the wrong half, because the up-front column is where the larger number usually lives.

In this TestMu Conf 2026 session, Cloud Assumes You Know What a Request Will Cost, Ojus Save states the same inversion from the infrastructure side: cloud was built around a request whose cost you can estimate before it runs, so you size timeouts to it, forecast capacity from its average, rate-limit against it, and bill by it, and an agent breaks that at the source because it decides what work to do while it executes.

What I Measured

Four flows, each stated as a single plain-English objective, run headless in agent mode on Kane CLI 0.8.4 on 27 August 2026. Each run reports its own duration in the terminal event that closes the stream.

kane-cli run --agent --headless "<objective>" --url <target> | tail -1 \
  | jq -r '[.status, .duration, .total_runs, .bifurcated] | @tsv'

# the run_end event carries status, duration in seconds, how many attempts
# the run needed, and whether the agent had to branch to reach the goal

The limits of this sample deserve stating before the numbers rather than after them.

• Four runs, not four hundred - enough to establish a range, nowhere near enough to establish a distribution.
• Small public pages - the TestMu AI playgrounds, with no authentication step and no heavy application shell to load.
• All passing paths - a failing run behaves differently, because the agent spends time trying alternatives before giving up.
• One machine, one session - no parallelism, no cold-start penalty from a fresh CI runner.

Read the result as a floor for a single flow, not as a forecast for your suite.

The Numbers

The four flows took 29.8, 31.2, 41.6, and 47.0 seconds, a mean of 37.4 seconds and a spread of 17.2 seconds between fastest and slowest. All four passed on the first attempt with no branching.

The slowest run is the most interesting one. Proving something did not happen took roughly half again as long as proving something did, because absence has to be established rather than observed.

A scripted equivalent of any of these would execute in single-digit seconds. That gap is real, and it is the price of not having written the script.

Where Plain CI Costs More

None of the following appears on a compute invoice, and together they usually exceed it.

• Selector maintenance - one renamed class turns dozens of specs red at once, and the week goes on repairs rather than on new coverage.
• Reruns on flake - every retry is compute you paid for that produced no information about the product. Scoped retries reduce that waste; TestMu AI's HyperExecute, for instance, retries only the failures whose logs match patterns you list.
• Quarantine drift - a test skipped in March is still skipped in September, and the coverage number never reflected it.
• Authoring latency - a flow with no test has no cost at all until it breaks in production, at which point it has all of the cost.
• Debugging archaeology - reconstructing a failure from a stack trace, a video, and screenshots in three systems, most of which expire.

The fourth item is the one that distorts every comparison. A suite covering forty percent of your flows looks cheap precisely because it is not covering the other sixty.

Where an Agent Run Costs More

The honest column, and it is not short.

• Every execution costs something - inference is metered, so a check running on every commit to every branch adds up in a way a scripted assertion does not.
• Wall-clock time is longer - tens of seconds against single-digit seconds, which matters when a pull request runs thirty of them.
• Timing varies between runs - the seventeen-second spread in four runs means capacity planning works on a range rather than a constant.
• A vague objective wastes a whole run - an ambiguous instruction produces an expensive result nobody can act on.
• Parallelism is not free either - running ten objectives at once shortens the wall clock and does nothing to the total.

The practical consequence is that agent-run checks belong on the flows that matter most, on the events that matter most, rather than everywhere by default. For how replaying passed flows and delegating the browser loop change that math, see agentic testing in the E2E stack.

The Cost Nobody Bills You For

There is a third column neither invoice contains: the defects that reached users because nothing checked the flow at all.

That cost lands as support load, churn, and an incident channel at ten at night. It is the largest number in the comparison and the only one nobody reports.

It also moves in the opposite direction to the visible costs. Every flow you leave uncovered makes the compute bill look better.

Which is why the right comparison is not cheaper against dearer. It is what each approach lets you cover at all, a framing we applied to a specific migration in the 30-day agentic end-to-end testing playbook.

The Evidence a Metered Run Has to Leave Behind

A metered run can spend its full inference bill and still not settle whether a criterion held. That gap widens when the system under test is itself an agent, because the agent writes its own account of the work, and an account is not proof that the work happened.

• Observed effects over summaries - TestMu AI's Agent Assurance grades each criterion against files changed on disk, artifacts produced, and tool calls checked against the agent's declared tool surface.
• Grey-box discovery - it reads manifests, prompts, tool tables, and MCP servers, and where an agent declares nothing it says so rather than guessing.
• Three verdicts - Pass, Fail, and Unable to Verify, with the last excluded from the pass-rate denominator so it is neither a soft fail nor a silent pass.
• The assurance gap - the share of criteria a run could not verify, reported beside the pass rate.

That last figure is the line to carry into your cost model, because it is the part of the inference bill that bought no information.

The terminal version, rook, is pre-alpha, and a finished run exits 0 whether the scenarios passed or failed. A pipeline gate therefore has to read the per-criterion verdicts rather than the exit code. The autonomous agent category, meaning agents that act rather than talk, is on a waitlist today.

Modelling Your Own Numbers

Do not adopt my four numbers. Run the same measurement on your own application, where the authentication step, the application shell, and the data volume are all yours.

• Pick five flows you would genuinely mind breaking, and write one objective for each.
• Run each ten times against a stable environment and record duration, attempts, and outcome from the closing event.
• Count what the equivalent scripted specs cost you last quarter, including repair hours and reruns, not just execution minutes.
• Add the flows you have no coverage for at all, because that is the column the whole exercise exists to expose.

Step three is where the numbers usually surprise people. Repair hours rarely get logged against testing, so they turn up as ordinary engineering time and vanish from the comparison.

Once you have a figure you trust, decide placement rather than replacement. The setup for running the command in a pipeline is in the Kane CLI introduction documentation, and the gate it plugs into is covered in a practical quality gate for AI-built pull requests.

Cloud Assumes You Know What a Request Will Cost and How Startups Are Rethinking Value and Monetization, both from Testμ 2026, approach this from different angles.