Author End-to-End Tests in Natural Language with KaneAI
Plan, Author, and Evolve E2E Tests for Web and Mobile Apps in Natural Language.
AI Regression Testing: Selection, Self-Healing, and Triage
AI regression testing uses AI to choose which tests run, write and repair them, and sort failures. See how each step works and where self-healing misleads.
Last Updated on:
Airbnb's engineering team used an LLM-driven pipeline to migrate nearly 3.5K React component test files from Enzyme to React Testing Library while keeping the original test intent and code coverage. The team had estimated 1.5 years of engineering time to do it by hand and finished in 6 weeks, according to Airbnb Engineering.
Keeping an existing suite current is one job in AI regression testing; choosing which tests a change needs and sorting failures by likely cause are the others. Each saves engineering time, and each can hide a real regression when nobody checks its output. A self-healing run on the TestMu AI grid showed both outcomes on the same Search button.
Overview
AI regression testing is regression testing in which AI picks the tests a code change needs, writes and updates those tests, repairs broken locators while the suite runs, and groups failures by likely cause. It cuts the upkeep of a large suite, but an engineer still reviews what each heal or rewrite changed.
Which Jobs Does AI Take Over in Regression Testing?
- Change-aware selection: KaneAI's GitHub App reads a pull request's diff and description, then adds semantically similar existing test cases from Test Manager to the run alongside the tests it generates.
- Natural-language authoring: KaneAI turns plain-English steps, Jira tickets, or a pull request diff into test cases and exports them to Selenium, Playwright, Cypress, or Appium code.
- Self-healing locators: TestMu AI Auto Healing re-anchors a locator that stopped matching. In a TestMu AI grid test, a renamed Search button healed in under two seconds and the search still ran.
- Parallel execution: HyperExecute runs a regression suite in parallel up to 70% faster than traditional grids, and its fail-fast threshold stops a job after a set number of consecutive failures.
- Failure triage: Test Intelligence ranks flaky tests by flake rate, groups failures by error message, and returns an AI root cause that engineers confirm in the logs before acting on it.
- KaneAI cost: There is no free KaneAI plan. Starter costs $19 per agent per month for local authoring through Kane CLI, and cloud web authoring starts on Pro at $99 per agent per month.
- Self-healing can pass a broken feature. In a TestMu AI grid test, a Search button detached from its form healed and clicked without an error while no search ran, and only an assertion on the results page failed the test.
What Is AI Regression Testing?
AI regression testing applies machine learning and large language models to the regression cycle: deciding which existing tests a change needs, generating and updating test cases, repairing tests at run time, and analyzing failures. The goal stays the same as in any regression run, which is proving that a change did not break what already worked. AI changes how much of that work a person does by hand.
You can add it as intelligent test automation features on top of an existing Selenium, Playwright, or Cypress suite, so adopting it does not have to start with a rewrite. Regression is one area among several; AI in software testing covers the rest of the lifecycle.
The term has a second meaning. Teams shipping LLM features, chatbots, or agents also say AI regression testing when they check that the AI itself did not get worse after a model, prompt, or tool change. That kind of check needs scored evaluations instead of exact assertions, and the section on regression testing an AI application covers it.
How Does AI Regression Testing Differ From Traditional Testing?
In a traditional suite a person makes each of these calls. In an AI-assisted suite a model makes the first pass and a person reviews it.
| Job | Traditional Regression Testing | AI Regression Testing |
|---|---|---|
| Test selection | Runs the full suite, or a subset someone picks from memory of what the change touched. | Picks or ranks tests from the diff, past failures, and similar existing tests, then a person checks what was left out. |
| Maintenance | Testers fix broken locators and steps by hand after each UI change. | Self-healing re-anchors a broken locator at run time, and the heal still needs review. |
| Test creation | Engineers write and update scripts in code, one flow at a time. | An agent generates tests from plain-English steps, tickets, or a pull request diff, then exports framework code. |
| Failure triage | Engineers read logs run by run to separate product bugs from flaky tests. | Failures are grouped by error and given a likely root cause that an engineer confirms. |
What Does AI Do at Each Step of a Regression Cycle?
Each step pairs what AI handles with the check that stays with a person. For a vendor comparison instead of a workflow, the regression testing tools roundup lists tools by category.
- Choosing what to run - Change-based selection maps what a commit touched to the tests that cover it, so a pull request runs a slice of the suite and the nightly job runs all of it. The KaneAI GitHub App integration reads a pull request's diff, title, and description, then adds semantically similar existing cases from TestMu AI's test management platform to the run. Check which tests a selection left out before trusting a green pull request; agentic regression testing covers how to verify a skipped test.
- Writing and updating tests - KaneAI, TestMu AI's GenAI-native testing agent, turns plain-English steps, Jira tickets, PRDs, or a GitHub pull request into test cases, and exports them to Selenium, Playwright, Cypress, or Appium so the suite stays in your framework. Review the generated assertions, since a test that clicks through a flow without checking the result only adds runtime. Prompting tips are in how to generate test cases with AI.
- Keeping tests running - Self-healing repairs a test when the page changes under it. Locator healing re-anchors a selector that stopped matching, while KaneAI's Adaptive Heal re-authors the failing objective and saves the result as a new test version that waits for approval by default, per the KaneAI healing documentation. Review each heal and each re-authored version before accepting it; self-healing test automation explains the mechanics.
- Checking what changed on screen - Visual regression compares each screenshot with an approved baseline. SmartUI, TestMu AI's visual testing tool, computes the pixel difference, filters rendering noise such as anti-aliasing and dynamic content, and classifies what remains for a reviewer to approve or reject; a rejected change blocks the build. Approving every diff turns regressions into new baselines, so review each one.
- Running the suite at scale - Parallel execution keeps a full suite inside its nightly window. HyperExecute splits tests across parallel machines and can stop a job once a set number of tests fail in a row, so a broken build stops consuming parallel capacity. For mobile apps, Real Device Cloud runs the suite on 50,000+ real Android and iOS devices, and HyperExecute parallelization for mobile tests covers the setup. Recheck the run time whenever the suite grows, so the full run still fits its window.
- Sorting failures - A red nightly run can mean a product bug, a flaky test, or a broken environment. Test Intelligence flags tests whose results change across runs and ranks them by flake rate, groups similar failures by error message, and its AI root cause analysis reads network, console, and Selenium logs to suggest a likely cause; for a failed HyperExecute job, the analysis comes back in 20 to 30 seconds. Treat the suggestion as a lead and confirm it in the logs before filing the bug.
Can Self-Healing Hide a Real Regression?
Yes. A heal only proves that the test found an element; whether the feature still works is a separate check. To see the difference on a real page, I ran one search test against three fake releases of the Ecommerce Playground demo store with Auto Healing enabled on TestMu AI's Automation Cloud.
Each release changes the desktop Search button before the test clicks it: a cosmetic class rename, a copy detached from its form so it no longer submits, and a copy whose type changed from submit to button. The test first locates the button once, because the Auto Healing documentation says a heal works from the DOM path recorded when an element was last found. It then applies the release, types iPhone, clicks the button through the old locator, and asserts that the results URL contains search=iPhone.
The script uses selenium-webdriver 4.51.0, which needs Node.js 22 or later, and ran on Chrome 154 on Windows 11 in build 108182621. Set LT_USERNAME and LT_ACCESS_KEY, save the three parts below in order as heal-check.mjs, and run node heal-check.mjs.
// heal-check.mjs, part 1 of 3: grid capabilities for an autoHeal session
import { Builder, By, until } from "selenium-webdriver";
const caps = { browserName: "Chrome", browserVersion: "latest" };
const options = {
username: process.env.LT_USERNAME,
accessKey: process.env.LT_ACCESS_KEY,
platformName: "Windows 11",
build: "AI regression: self-healing audit",
autoHeal: true,
};// heal-check.mjs, part 2 of 3: three fake releases and a session helper
const copy = "c = b.cloneNode(true); c.className = 'search-v2';";
const releases = {
"cosmetic rename": "b.className = 'search-v2';",
"detached from form": `${copy} c.setAttribute('form', 'x');`,
"type set to button": `${copy} c.type = 'button';`,
};
const open = async (name) => {
const driver = await new Builder()
.usingServer("https://hub.lambdatest.com/wd/hub")
.withCapabilities({ ...caps, "LT:Options": { ...options, name } })
.build();
return { driver, id: (await driver.getSession()).getId() };
};// heal-check.mjs, part 3 of 3: the regression check, once per release
for (const [name, change] of Object.entries(releases)) {
const { driver, id } = await open(name);
let ran = false;
try {
await driver.get("https://ecommerce-playground.lambdatest.io/");
await driver.findElement(By.css("button.type-text")); // baseline
await driver.executeScript("let c; const b = document.querySelector" +
`("button.type-text"); ${change} if (c) b.replaceWith(c);`);
await driver.findElement(By.name("search")).sendKeys("iPhone");
await driver.findElement(By.css("button.type-text")).click(); // heals
await driver.wait(until.urlContains("search=iPhone"), 10000);
ran = true; // the outcome assertion: the search really ran
} catch (error) { console.log(`${name}: ${error.name}`); }
await driver.executeScript(`lambda-status=${ran ? "passed" : "failed"}`);
await driver.quit();
console.log(`${name}: search ran = ${ran} (${id})`);
}This is the console output of one run. Four more runs gave the same three results: three made while the script printed session IDs in a different format, and one made from the code exactly as printed on this page.
cosmetic rename: search ran = true (5592c3481d233347f89698cb380f8e19)
detached from form: TimeoutError
detached from form: search ran = false (d8886ac8a900e82f20216fdf850ef158)
type set to button: ElementNotInteractableError
type set to button: search ran = false (18350d7c7b1b3c12cdc309230e403fcb)The test result alone does not show what the healer did. Each healed lookup is recorded in the session's command log, and heals.mjs reads that log through the TestMu AI REST API with the same credentials. Run it with the session IDs from the output above, as in node heals.mjs followed by the three IDs.
// heals.mjs: print every locator the grid healed in finished sessions
const token = Buffer.from(
`${process.env.LT_USERNAME}:${process.env.LT_ACCESS_KEY}`,
).toString("base64");
const api = "https://api.lambdatest.com/automation/api/v1/sessions";
for (const id of process.argv.slice(2)) {
const res = await fetch(`${api}/${id}/log/command`, {
headers: { Authorization: `Basic ${token}` },
});
const { data } = await res.json();
for (const step of data.filter((entry) => entry.autoHeal)) {
const asked = JSON.parse(step.Value.requestBody).value;
const healed = JSON.parse(step.healedLocator).value;
const ms = step.Value.duration;
console.log(`${id.slice(0, 8)} ${asked} -> ${healed} (${ms} ms)`);
}
}5592c348 button.type-text -> button.search-v2 (1638 ms)
d8886ac8 button.type-text -> button.search-v2 (1393 ms)
18350d7c button.type-text -> button.type-icon (1190 ms)Put side by side, the healer made a different choice for each release:
| Fake Release | Element the Healer Chose | Test Result | What Caught It |
|---|---|---|---|
| Cosmetic rename | The same Search button under its new class | Search ran and the test passed in all five runs | Nothing needed catching; the heal was correct |
| Detached from form | The broken copy of the button | Click succeeded, no search ran, and the test failed in all five runs | Only the assertion on the results URL |
| Type set to button | A hidden search button in the mobile header | Click failed with element not interactable in all five runs | The click error, because the chosen button was hidden |
The detached copy is the case to worry about. Without the final assertion, that test passes while no customer can search. The third release failed only because the button the healer picked happened to be hidden, and a visible lookalike would have taken the click.
Across all 17 sessions in the build, healed lookups took 0.68 to 3.23 seconds, against 21 to 55 milliseconds for the same lookup before the change. The build also holds two sessions from a first draft of the script, whose URL check looked for product/search and so failed even the correct heal, because the form encodes that slash as %2F. Checking the value typed into the search box fixed it.
Keep healing on only with these guards in place:
- Assert the outcome - End every regression check on the result a user sees, such as a results URL, a cart total, or a confirmation message, and never on the click.
- Read the heal log after each run - The command log records every healed lookup with the old and new locator, so a heal onto a different element shows up the same day.
- Turn healing off where any UI change must fail - The Auto Healing documentation warns that it "may also mask real issues in your web application or test scripts", so leave it off for a strict release gate such as checkout. It also cannot run in the same session as SmartWait.
Skip the setup and install the Selenium Skill for Claude Code, Copilot & Cursor with one command.
How to Use AI for Regression Testing
Google's 2024 DORA report found that AI adoption negatively impacts software delivery stability and throughput, and that "fundamentals like small batch sizes and robust testing remain crucial." For a five-level maturity model with measurable criteria for that adoption, see how teams use AI in quality assurance.
Roll AI into one suite at a time, starting where maintenance costs the most:
- Measure the suite first - Record its runtime, flaky-test rate, and the hours spent repairing tests each sprint, so you can tell later whether AI reduced any of them.
- Start with the suite you repair most - Turn on healing there first, and add an outcome assertion to every test that lacks one before you trust a healed pass.
- Select tests per pull request - Comment @TestMuAI Validate this PR on a pull request, and the KaneAI GitHub App generates tests from the diff and adds similar existing cases. Keep running the full suite on a schedule, since a selection can miss a test that an indirect dependency breaks.
- Run the full suite nightly - In Test Manager, open the project's Test Runs and set a date and time, as the KaneAI scheduled test runs guide shows; the schedule's advanced settings include self-maintenance, visual regression, and accessibility testing.
- Triage before you fix - Start each morning from the failure clusters and the flaky-test ranking instead of the raw list of red tests, and confirm every AI root cause in the logs.
- Review what the AI changed - Approve or reject re-authored test versions, scan the heal log, and keep a person's sign-off on every release.
Note: Run your regression suite with Auto Healing on TestMu AI and check every heal in the command log. Sign up for free
How Do You Regression Test an AI Application?
When the code under test is itself AI, a regression shows up as worse answers after a model, prompt, or tool change. Exact-match assertions break because the same prompt can return different wording on every run, so teams score a fixed set of inputs against criteria and fail the build when a score drops below a tolerance band.
- LLM features - LLM regression testing covers release gates that survive a model upgrade, including answer churn, cost, and latency as regression signals.
- Agents that take actions - Agent regression testing covers what breaks AI agents between releases, what to assert, and how to gate a release on the verdicts.
- Voice and chat agents - AI voice agent regression testing covers baselines and scoring for quality drops after a prompt, model, or flow change, and TestMu AI's Agent Testing grades chat, voice, and phone agents on what they say.
- GenAI features inside an app - KaneAI's AI app testing adds prompt assertion templates and LLM output validation to the same end-to-end run as the UI and API checks.
- Machine learning models - AI/ML testing covers accuracy degradation and behavioral drift across model versions.
How to Create an AI Regression Test With KaneAI
Cloud web authoring needs the KaneAI Pro plan or higher; Starter covers local authoring through Kane CLI. The steps below follow the KaneAI web test guide.
- Open KaneAI and select Create a Web Test.
- Write each step as a plain-English objective, such as "Search for iPhone" and "Check that the results page lists iPhone products". The guide notes that the more specific the objective, the more accurate the generated test case.
- Select Finish Test, pick the project and folder, keep or edit the test name and description KaneAI generates, and save the test case.
- Pick a healing strategy for the test, either Adaptive Heal, Dynamic Test, or Retry on Failure. The strategies apply to New Experience test cases on Chrome, one at a time, and the two that re-author the test consume credits.
- Add the test to a scheduled test run in Test Manager so it joins the nightly regression.
- Export the code when your team wants to own it: KaneAI generates Selenium, Playwright, Cypress, or Appium code from the saved test case.
For a no-code walkthrough built around a whole regression suite, see how to automate regression tests without code. To validate the skills hands-on, take the KaneAI Certification.
Conclusion
AI regression testing pays off first on the suite your team repairs most often: turn on Auto Healing there, add a results assertion to every test, and run heals.mjs after the first nightly run to see what the healer changed. To write new regression tests in plain English, start with the KaneAI getting started guide.
Testing web or mobile apps without writing code? KaneAI by TestMu AI turns plain-English instructions into tests.
Author
Shantanu Wali is Vice President of Product Management at TestMu AI, where he owns several product lines across the testing platform, including the Real Device Cloud and the Digital Experience Testing Cloud. He has also contributed significantly to the development and scaling of KaneAI, TestMu AI's flagship GenAI-native testing agent that uses natural language to make software testing faster and more reliable in this AI era. He brings 7+ years of experience across software development and product management, starting as a backend developer at Infosys building solutions for Fortune 500 clients. Shantanu holds an MBA from IIM Calcutta and a B.Tech in Mechanical Engineering.
Reviewer
Navin Chandra is a Member of Technical Staff at TestMu AI, building the open-source automation that powers its Selenium and Appium cloud grid. A committer to both Selenium and Appium, he implemented WebDriver BiDi support in Selenium for real-time browser events and bidirectional control and is developing Apple's iOS RemoteXPC protocol in Appium to enable low-level wireless communication with iOS system services. He contributes to Selenium across multiple language bindings as a member of the Selenium GitHub organization. He has served as a Google Summer of Code mentee and mentor at openSUSE and an LFX mentee at CNCF's KubeArmor, and is a SUSE Certified Deployment Specialist. Navin holds a B.Tech in Computer Science.
AI Regression Testing FAQs
Did you find this page helpful?
More Related Blogs
TestMu AI forEnterprise
Get access to solutions built on Enterprise
grade security, privacy, & compliance
- Advanced access controls
- Advanced data retention rules
- Advanced Local Testing
- Premium Support options
- Early access to beta features
- Private Slack Channel
- Unlimited Manual Accessibility DevTools Tests

















