AI Test Maintenance Without False Passes

Why Do Recorded Tests Go Stale?

A recorded test replays the steps it was authored with. The application does not hold still, and each kind of change breaks the replay in a different way and deserves a different repair.

The last row is the one automation cannot see from the inside. Every row above it looks identical to the row below at the moment of failure, which is the whole problem this page is about.

When Is a Passing Test Not a Working Test?

A repair is judged by whether the test passes afterwards. That is a weak oracle, and the research on automated repair has been saying so for a decade.

The most direct evidence comes from an industrial case study of an autonomous multi-agent testing system. Across 300 consecutive execution reports covering 636 test-case executions, Practical Limits of Autonomous Test Repair reports a 70% repair convergence rate at the scenario-family level, and a semantically strict rate of 50% once the families that converged by weakening an assertion or deleting a test case are excluded.

Twenty points of that headline number were repairs that made the test pass by making it test less. The study names the two mechanisms it observed.

• Assertion weakening - one family converged by relaxing a strict equality check into a truthiness check, which passes for almost any value the application returns.
• Silent test-case deletion - another converged because the failing case stopped existing, which removes the failure and the coverage together.
• Convergence took several attempts - only 10% of scenario families succeeded on the first attempt, at a mean of 4.4 repair iterations, so most repairs are the product of a loop rather than a single confident edit.

The same weakness shows up in program repair, where the test suite is the thing being satisfied. Work on identifying patch correctness in test-based program repair starts from the observation that test suites in practice are often too weak to guarantee correctness, and that existing approaches generate a large number of incorrect patches. Evaluated on 139 patches from five repair systems, its heuristic blocked 56.3% of the incorrect ones without blocking any correct patch.

Read across the two, the useful conclusion for a QA team is narrow and firm. Automated repair earns its place as a proposal engine, and something outside the loop has to decide whether the proposal is a fix or a cover-up.

What Exactly Gets Repaired, the Locator or the Test?

Vendors use "self-healing" for two mechanisms with very different risk profiles. Separating them is the first governance decision, because only one of them changes what your test asserts.

• Locator recovery is low risk by construction - it changes the address of an element, not the intent of the step. KaneAI's Auto-Heal tries the other locators captured for that element, then rebuilds one from the step's original natural language instruction, and creates no version.
• Re-authoring is a content change - it rewrites the objective, so the new steps and the new assertions are generated rather than authored by a person. That is why it produces a version and why the version waits.
• They can both act on one run - locator recovery runs first, and re-authoring is what handles the step it could not recover.

If a tool cannot tell you which of these two it is doing, treat everything it does as the second one. The reviewing effort you save by assuming the first is exactly the effort that catches a weakened assertion.

Two categories is the minimum. The useful version of the question is how much of the original test a system is allowed to reinterpret, and that runs to six levels.

Approval requirements should climb with that table rather than sit flat across it. The cheap deterministic levels earn their autonomy; the interpretive ones have not.

Order matters as much as permission. Running the expensive interpretive level first is how teams end up paying an agent to solve a problem a fallback locator would have solved, and research on locator repair has been arguing the same for years: Brisset, Rouvoy, Seinturier and Pawlak built Erratum on tree matching precisely because scanning every element on a changed page gets slower and less accurate as interfaces grow.

How Much of a Test Does One Failure Rewrite?

Re-authoring is not a surgical edit to the step that broke. The unit is the objective, and the scope runs to the end of the test.

That shape has three consequences worth planning around, and they all point the same way: a failure early in a long test is the expensive one.

A run in which nothing fails to replay does not trigger re-authoring at all, so the cost is a function of instability rather than a standing charge. The one exception is authoring from objectives on every run, which pays that cost whether or not anything broke.

Which UI Changes Should a Test Absorb?

Most healing policies are written in terms of confidence, which describes the system's certainty and says nothing about whether the change was one a test should follow. Classifying the drift instead gives you a rule that survives contact with a real interface.

The boundary between D3 and D4 is the one worth writing into policy. Above it the application moved and the test should follow; below it the application broke and following it produces the false pass.

• D0 and D1 are free - a renamed class or a moved node changes nothing a user could observe, and a repair here needs no ceremony beyond a log line.
• D2 is where agents start earning their cost - when a dropdown becomes a set of cards there is no old locator left to repair, so pattern matching has nothing to work with and only intent does.
• D3 needs a named reason - a new step in the journey is either a product decision somebody made or a bug, and a reviewer should be able to point at the release that caused it.
• D4 and D5 are findings - the correct output is a defect report and a red run, and any system that repairs its way past them is working against you.

A healer that reports only pass or fail cannot express this distinction, which is the practical argument for carrying the drift class into the run result rather than leaving it in a log.

What Should a Repair Never Be Allowed to Change?

The D3 to D4 boundary is only enforceable if the test states what it is proving separately from how it proves it. Splitting the two turns a cultural norm into something a system can check.

The practical form of the immutable half is a short list of user-visible facts that must still hold after any repair. For a shopping journey it is close to trivial to write down.

• The product identity - the item in the basket is the item the objective named, not the one nearest the pointer.
• The chosen variant - size M stays size M. A repair that settles for L because L was the easiest chip to find has changed the test.
• The quantity - one remains one, which catches the class of repair that clicks twice to make a stubborn control respond.
• The visible confirmation - the basket badge and the line item still appear, so success is measured on the screen rather than on a method returning without an exception.

Checked after the repair rather than before it, that list is the difference between a test that adapted and a test that gave up. It also gives a reviewer something concrete to disagree with, which a confidence score never does.

Confidence and risk belong on separate axes for the same reason. Being 99% certain that a control is the delete button is not an argument for clicking it, and a policy that folds one number into the other loses exactly the distinction that matters on destructive actions.

The third row is the one that separates a governed system from an eager one. It is also the row most healing configurations do not have, because a single threshold cannot express it.

What Does One Repair Actually Look Like?

Everything above is policy. This is the same policy applied to one checkout test across two releases, where the interface change is identical and the correct verdict is opposite.

The test has five objectives. The third selects a delivery option.

objective_3: Choose express delivery

recorded_steps:
  - click:  "#delivery"            # a <select> element
  - select: "express"
  - assert: text "Express delivery" is visible

invariants:                        # what this objective proves
  - selected_delivery == "Express"
  - delivery_cost is visible
  - basket_total updates

Release A: the checkout is redesigned

A designer replaces the dropdown with three cards. No selector in the recorded steps resolves, the step fails, and re-authoring produces this draft.

  objective_3: Choose express delivery

- click:  "#delivery"
- select: "express"
+ click:  card labelled "Express delivery"

  assert: text "Express delivery" is visible

Run the four reviewer checks against it and every one passes: the assertion is unchanged, nothing was dropped, the failure has a matching design ticket, and the rewrite is confined to the control that moved. The invariants hold after the repair, because Express is still what ends up selected. Approve.

Release B: the same failure, a different cause

A pricing change removes Express from three postcodes, and a bug removes it from all of them. The step fails exactly as before, and the draft comes back like this.

  objective_3: Choose express delivery

- click:  card labelled "Express delivery"
+ click:  card labelled "Standard delivery"

- assert: text "Express delivery" is visible
+ assert: a delivery option is selected

The run goes green. The suite reports no regression. Two of the four checks fail: the assertion was weakened from a specific string to an existence check, and no ticket explains why Express disappeared. The invariant selected_delivery == "Express" is false. Decline, and raise the bug.

Nothing in the run distinguishes these two. Same failing step, same green result, same draft waiting in Version History. The only thing separating a legitimate repair from a shipped bug is a person reading the second diff against a list of facts written before either release.

This is also the argument for writing the invariants down first. A reviewer looking at Release B without them sees a plausible rewrite of a test they did not author, under time pressure, in a queue of other drafts.

Which Strategy Fits Which Suite?

The choice is usually made once, at project level, for suites that have nothing in common. Matching the strategy to what the suite is for produces a better answer than a single organization-wide default.

Retry on Failure deserves one caution that is easy to miss. Retrying is the right answer only when the failure is genuinely non-deterministic, and using it to paper over a real defect produces the same false pass by a slower route, which the flaky test guide covers in more depth.

What Should a Reviewer Actually Check?

"Review the draft" is not a process until someone writes down what would make them decline one. These five checks come straight from the failure modes the research documented.

• Does it still assert the same thing? Compare the draft against the current version and read the assertions first, not the steps. An equality check that became a presence check is the documented failure mode.
• Did anything get dropped rather than fixed? A shorter objective that passes is a coverage loss wearing the costume of a repair.
• Is there a defect report for the original failure? If the step failed because the product broke, the correct outcome is a declined draft and a raised bug, not a new version.
• Does the rewritten portion match a real product change? Someone should be able to name the release or ticket that made the old steps wrong.
• Is the rest of the rewrite incidental? Objectives after the failure are rewritten because of their position, so changes in them need the same reading even though nothing in them failed.

KaneAI supports this with the mechanics rather than leaving it to discipline.

• The draft is attributed to the strategy - not to the person who started the run, so nobody inherits authorship of a change they did not make.
• One test case produces one draft - a run covering the same case under several browser versions or operating systems still yields a single draft to read, because it is one test case.
• A waiting draft cannot be edited - it has to be approved or declined first, which stops a half-reviewed rewrite from being patched into acceptability without a verdict.
• Approving regenerates the exported code - so the framework code your pipeline runs matches the version that was approved rather than the one before it.
• Declining costs only the run you already paid for - it discards the draft and leaves the current version untouched, which makes declining the cheap option rather than the disruptive one.

How Does a Good Repair Become a Bad Baseline?

A repair that worked once worked under the exact conditions that produced the failure. Promoting it on that evidence is how a lucky recovery becomes the reference point every later run is measured against.

The failure mode compounds in any system that learns from successful runs, and none of the steps look wrong on their own.

• A repair picks the wrong but plausible target.
• The step completes, so the run is recorded as successful.
• That successful run becomes the new reference for what normal looks like.
• Later repairs measure themselves against the wrong target and agree with it.
• The original defect is now invisible, and the suite is confidently wrong.

The fix is not to stop learning from successful runs, which would throw away the thing that makes healing cheap. It is to make promotion conditional on the user-visible facts rather than on the absence of an exception.

The same logic is why a run that passed after a repair should not be filed under the same result as a run that passed outright. Collapsing the two hides the number a team most needs to watch: how much of the suite is now standing on repairs nobody read.

Where Should the Strategy Be Decided?

A maintenance strategy set once at the top of an organization is a policy decision disguised as a toggle. The inheritance rules decide who actually owns it.

• Set the organization level to the safest option - it is a default for projects that have not thought about it yet, and the safest default is the one that fails loudly.
• Expect projects to diverge and let them - a project follows the organization until someone changes it there, after which it keeps its own value and later organization changes no longer overwrite it.
• Treat run-level choices as experiments - a run starts from the project value and can override it, and changing it in a run never writes back to the project or the organization.
• Remember it is not retroactive - turning self-maintenance on applies from that point forward, and runs that already exist are not eligible whatever the setting now says.

One practical habit prevents most surprises. Confirm the strategy in Advanced Configurations before executing, because that is where the organization, project and run-level values resolve, and the test run panel shows the resolved strategy alongside the instance count and concurrency.

How Do You Know the Gate Is Working?

The obvious metric is the wrong one. A rising pass rate after enabling automated repair is exactly what you would see if the repairs were correct, and also exactly what you would see if they were hiding defects.

The second row is the one to put in front of a stakeholder. It converts a governance step that looks like overhead into a count of defects that would otherwise have shipped, which is the argument for keeping approval switched on.

One derived measure is worth more than the rest of the table combined. Call it the false-green rate: repaired passes that no longer prove what the test was written to prove, divided by all repaired passes.

It reframes what a good result looks like. A system recovering 98% of broken tests with a 5% false-green rate is less trustworthy than one recovering 90% with almost none, because the first is retiring coverage while reporting success.

Give the pipeline more than two words for what happened

• Passed outright - the recorded steps replayed and nothing was repaired. This is the only result that needs no further thought.
• Passed after a retry - nothing in the test changed, but the count belongs in the result, because a step that needs three attempts is reporting something about the product.
• Passed after a locator repair - safe to ship, worth counting, and a rising trend says the interface is churning faster than the suite is being maintained.
• Passed after a re-plan - the journey changed shape. This is the state that should never be collapsed into a plain pass.
• Held for review - the repair was plausible but crossed a risk threshold, so the run has an answer and a person still owes it a verdict.
• Failed on meaning - a repair was possible and was refused because it would have changed what the test proves. This is the system working correctly.

The last state is the one most tools cannot express, and it is the one that makes a healing system trustworthy. A refusal to repair is a finding, and a pipeline that can only say pass or fail has nowhere to put it.

Where to set the first thresholds

None of these are benchmarks, because no published dataset covers them. They are starting lines, chosen so that crossing one prompts a conversation rather than an alarm, and meant to be replaced by your own numbers after a quarter.

The second row is deliberately counterintuitive. A gate that has never declined anything is not evidence of a healthy suite, and treating zero as a good score is how a review step becomes a rubber stamp nobody notices.

What Changes by Team and Domain?

The policy is the same everywhere. What moves is how expensive a wrong repair is, and that should decide the settings rather than the size of the team.

The last row is the one small teams get wrong most often. Facing a review queue they cannot service, the instinct is to turn auto-approve on, which removes the gate precisely because there was no capacity to staff it.

Narrowing what is eligible is the better trade. Ten tests under repair with every draft read beats two hundred under repair with none, because the second arrangement produces confidence without evidence.

How Does an Agentic Repair Loop Actually Work?

Once repair moves past locator substitution it stops being a lookup and becomes a loop: try something, look at what happened, try again. What separates the implementations is not the reasoning, it is where the loop is made to stop.

Playwright's test agents document the shape openly. Its healer replays the failing steps, inspects the current UI for equivalent elements or alternative flows, suggests a patch such as a locator update, a wait adjustment or a data fix, and re-runs until the test passes or guardrails stop the loop.

One detail in that description is worth more than the rest. If the healer concludes the underlying functionality is genuinely broken, it skips the test rather than continuing to repair, which is the same refusal the drift envelope asks for, expressed as a product behaviour.

The refusal row is where most evaluations stop short. A healer that skips a test it cannot honestly fix has produced a finding, and a suite that carries skipped tests without surfacing them is not much better than one carrying false passes.

What the documented approaches actually repair

Read down the third column rather than the first. Approaches that leave nothing behind cannot produce a false pass that outlives the run; approaches that leave a version or a patch can, and those are the ones that need the gate.

Several vendors publish less than this about where their repair stops, which is itself an evaluation signal. A tool that will not tell you its unit of repair or its refusal behaviour cannot be governed with the policy above, whatever its recovery rate looks like in a demo.

Both belong in the result rather than in a log, which is the same argument as testing agentic AI beyond pass fail. For how agents fit into UI automation more broadly, agentic testing in UI automation covers the surrounding workflow.

What Happens When the UI Is Rewritten Every Prompt?

Everything above assumes a UI that changes at human speed. Front ends built and revised through coding agents break that assumption, and they are where the distinction between structural and semantic drift stops being academic.

A single instruction such as "make the checkout cleaner and turn the delivery options into cards" can restructure the interaction model while leaving the user's goal untouched. The test is now structurally wrong and semantically correct, which is a combination traditional selector repair has no answer for.

That last row is the reason the intent layer matters more here than anywhere else. When a generated interface drops an option the user never asked to remove, the only thing standing between that and a green suite is a test that refuses to select the nearest alternative.

Tests written as natural-language objectives have a structural advantage in this setting, because an objective survives a rewrite that invalidates every selector under it. The Kane CLI runs those objectives from the terminal and emits a structured event stream a coding agent can read, which makes UI verification something the agent loop can consult rather than something a human runs afterwards.

Does Any of This Change on Mobile?

The governance holds. What changes is how a target is identified and how many ways a step can fail without the interface having drifted at all.

TestMu AI's Smart Heal shows the shape of a mobile healer. It needs at least one passed run to capture a baseline snapshot of every locator in the script, compares element attributes, hierarchy and visual cues when a later locator fails, retries the step with the healed locator, and logs the original alongside the recovered one. Where it cannot identify an alternative confidently, it records a suggestion instead of forcing a heal.

One documented behaviour deserves the attention of anyone running it: the baseline updates automatically after each successful run. That is efficient, and it is the exact mechanism the poisoning section describes, which makes the invariant check before promotion a practical safeguard rather than a theoretical one.

Element not found is not one failure

• Not rendered yet - the screen is still settling, which is a bounded wait rather than a locator problem.
• Off-screen - the control exists below the fold, so the answer is a controlled scroll.
• In another window - elements belonging to a keyboard or overlay may sit outside the default hierarchy entirely.
• The hierarchy moved - the genuine locator-healing case, and the only one of these five that a healer should repair.
• The control is gone - a finding, and the case where healing is the wrong instinct.

Sending all five to an AI matcher produces repairs for problems that were never about identity. Localisation makes the point sharpest: Appium's XCUITest locator strategies warn that an element's name attribute "is commonly rendered in the application GUI, which means it is likely to change depending on the application language".

A button whose label moves from Continue to Continuer has not disappeared. A healer that treats the visible label as identity will either fail a correct build or heal onto whatever else is nearby, and both outcomes are worse than the locale switch that caused them.

How Would You Prove a Healer Is Safe?

Vendor demos ask whether the test recovered. The question that decides whether you can trust a healer is whether it recovered the right behaviour, and that needs a test suite of its own: deliberate changes to your interface, each with a known correct outcome.

Two families, and the second is the one everybody skips.

A benchmark built only from the first family makes an over-eager healer look excellent, because every mutation in it rewards repair. The second family is where a healer earns trust, and refusing correctly should score as a pass rather than a miss.

Eight mutations against one representative journey is enough to rank tools, settle an internal argument about auto-approve, or give a team the evidence to raise a threshold it had been guessing at.

Where Does This Approach Stop?

Automated re-authoring covers a narrower slice of a suite than the phrase "self-healing tests" suggests, and the boundaries are worth knowing before a rollout plan depends on them.

• Eligibility is narrow - in KaneAI, re-authoring applies only to test cases using New Experience with a Chrome browser configuration, and everything else replays its recorded steps whatever is set.
• One strategy at a time - selecting one clears the others, so a run cannot both re-author and retry, and a suite needing both behaviours needs two runs.
• It does not reduce the review workload to zero - it converts a rewrite into a review, which is a significant reduction in maintenance rather than the removal of it.
• It repairs tests, not test design - an objective that was ambiguous when a person wrote it will be re-authored into something equally ambiguous.
• A stale repository is still stale - repair keeps individual tests running and does nothing about the duplicates and dead cases around them, which is the separate problem covered in duplicate test case detection with AI.

The honest framing for a rollout is that this replaces one kind of work with a smaller kind of work. Teams that budget for the review time get the benefit, and teams that assume the review disappears are the ones who find the weakened assertions months later.

The ways a healing programme goes wrong, and what stops each one

Each row has the same shape: something cheaper than the correct answer was available, and the system took it. That is what a budget and a review gate exist to interrupt.

Conclusion

Pick the one suite that costs the most to maintain, set Adaptive Heal on a single run with approval left off, and read the first draft it produces against the current version before approving it. That single review tells you more about whether automated maintenance suits your application than any pass-rate chart will.

From there, keep approval on wherever a wrong repair is expensive, and start counting declined drafts that turned into defects. The Adaptive Heal and Dynamic Test documentation covers the settings, and the feature walkthrough covers the screens they live on.