Intent-Based Testing: What It Is and How It Works

What Intent-Based Testing Actually Is

Intent-based testing states the outcome a test must prove and delegates the route to an agent. The test says a user can complete checkout and receive a confirmation. It does not say which button carries which id, or how many divs sit between the form and the submit control.

The reason this matters now is a change in where code comes from. Google's DORA research finds that 90% of survey respondents report using AI at work, and that AI adoption continues to have a negative relationship with software delivery stability. The 2025 DORA report announcement names the control that decides which way it goes: without robust control systems, "like strong automated testing, mature version control practices, and fast feedback loops, an increase in change volume leads to instability."

That is the pressure. Code volume is rising, the UI churns faster, and a suite whose tests are pinned to DOM paths gets more expensive exactly when it needs to get cheaper. The practical question is what a test should be attached to so it survives.

Intent-Based vs Script-Based Testing

What separates the two approaches is the input format far less than what the test breaks on.

Mainstream frameworks already moved partway here, which is the part most write-ups on this topic miss. The Playwright locators documentation states that "CSS and XPath are not recommended as the DOM can often change leading to non resilient tests," and recommends role-based locators because a role "reflects how users and assistive technology perceive the page."

A role locator is an intent-flavored locator. It names what the control is for rather than where it sits. Intent-based testing extends that idea from the element up to the whole assertion. If you are still writing selector-first suites, the Playwright tutorial covers the locator strategies this builds on.

How an Intent Becomes an Execution

Between the sentence you write and the browser action there are four steps, and knowing them is what lets you debug a run that goes wrong.

• Intake - The intent arrives as a typed sentence, a requirement document, a ticket, a recording, or a pull request diff. KaneAI accepts all of these and parses them into structured scenarios rather than starting from a blank test file.
• Plan generation - The intent is expanded into ordered steps with assertions proposed where validation matters. This plan is human-readable and reviewable before anything executes, which is the first place to catch a misread requirement.
• Element resolution - Each step is resolved against the live page by role, accessible name, and surrounding context instead of a stored selector. This is the step that makes the test survive a refactor, and it is the step covered by the measurement in the next section.
• Verdict - The run produces a pass only when the expected state is confirmed through evidence: DOM state, accessibility labels, a URL change, a network response, or an assertion you defined.

Step four is where most of the engineering argument lives, and it is worth being precise about it. The model is not deterministic. The contract can be. A language model reasoning about a page may take a different internal route on each run, so the click path is not guaranteed to be identical. What stays fixed is the rule for granting a pass: the agent decides how to reach an element, and does not get to decide on its own that the test passed. The KaneAI command guide documents the command vocabulary this resolves into.

Those four steps collapse into three layers once a test is running, and separating them is the fastest way to triage a failure. A red run means one specific layer disagreed with you, and each layer fails for a different reason.

The layer split also explains how mature implementations keep model latency out of the common path. Resolve the intent once, cache the element it resolved to, and replay the cached target on later runs; when the cached target stops matching, fall back to resolving again and heal it. Normal runs then cost about what a scripted run costs, and the model is only invoked at the moment something actually moved.

Which Locators Actually Survive a Refactor

The claim that semantic resolution survives UI change is repeated constantly and rarely measured. So we measured it.

Method. We loaded the Simple Form Demo page on the TestMu AI Selenium Playground in a real Chrome session on TestMu AI Browser Cloud and resolved one target control, the "Get Checked Value" button, three ways: by CSS id, by absolute XPath derived from the live DOM, and by role plus accessible name. We then applied a routine front-end refactor to the live page, renaming the id from showInput to submit-message-cta, replacing the class list, and wrapping the button in a new div. Then we resolved all three again. The refactor was applied deliberately in-page so both passes ran against the same page in the same session; it is a controlled simulation of a rename, not an observed production incident.

The three locators, exactly as run:

await page.goto('https://www.testmuai.com/selenium-playground/simple-form-demo');

// 1. Implementation-coupled: bound to an id attribute
await page.locator('#showInput').first().waitFor({ state: 'attached', timeout: 5000 });

// 2. Structure-coupled: bound to a position in the DOM tree
const xpath = '/html[1]/body[1]/div[1]/div[1]/main[1]/div[1]/section[2]' +
              '/div[1]/div[1]/div[1]/div[1]/div[2]/div[1]/div[1]/button[1]';
await page.locator(`xpath=${xpath}`).first().waitFor({ state: 'attached', timeout: 5000 });

// 3. Intent-coupled: bound to what the control is for
await page.getByRole('button', { name: 'Get Checked Value' })
          .first().waitFor({ state: 'attached', timeout: 5000 });

Results from the run, with resolution time measured to the point the locator attached:

Three findings worth carrying forward. The rename alone was enough to break both implementation-coupled strategies, and neither failure had anything to do with whether the button worked. The role locator kept resolving because the button's purpose did not change, only its markup. And the intent-coupled lookup was not slower in the healthy case, resolving in 113 ms against 264 ms for the id selector, so the cost argument against semantic resolution does not hold at the element level.

One caveat, stated plainly: this measures element resolution, not a full agent run. An agent reasoning about a page adds model latency that a single locator call does not. What the test isolates is the binding, and the binding is the part that decided whether the test survived.

The False Pass Is the Real Risk

Script-based suites fail loudly and waste your time. Intent-based suites can pass quietly and waste your confidence, which is worse, because nobody triages a green run.

There is now measured evidence for how wide the gap between "the test ran" and "the test would catch a bug" can get. In Benchmarking LLMs for Unit Test Generation from Real-World Functions, Huang, Zhang, Harman and colleagues evaluated model-generated tests and reported that on the TestEval benchmark they reached 92.18% statement coverage but only a 49.69% mutation score. On their harder ULT benchmark, built from real-world Python functions, the average mutation score fell to 40.21%.

Mutation score measures whether a test actually detects deliberately introduced faults, which is what makes the pairing Huang and colleagues report so pointed: near-total statement coverage sitting beside a mutation score roughly half as large describes tests that execute the code thoroughly and notice very little about it. Their study covers generated unit tests rather than browser intents, and the mechanism transfers, because when the thing being generated is the check itself, running is not the same as verifying.

In practice the false pass has one common cause, and it is the objective, not the model. An instruction like check that the page works defines no pass condition. The agent loads the page, observes it, finds nothing that contradicts a vague goal, and reports a pass that carries no information. The same run with a named piece of evidence attached would have had something to fail against.

Writing Intents That Can Actually Fail

A usable intent combines three things: an action to perform, an assertion that can be falsified, and an extraction when a downstream step needs the value. Assertions are the load-bearing part, because an intent with no assertion cannot fail for the right reason.

# Weak: no defined pass condition, so a pass means nothing
kane-cli run "go to the checkout page and check that it works"

# Strong: named evidence the run must produce
kane-cli run "go to https://ecommerce-playground.lambdatest.io/,
  search for 'iPhone',
  add the first result to the cart,
  open the cart,
  assert the cart contains 'iPhone',
  assert no error message is visible,
  store the cart total as 'cart_total'"

Assertion styles that give a run something concrete to fail against:

• Exact match - assert the total shows a specific value, used when the number is the behavior under test.
• Contains - assert the page contains a confirmation string, which tolerates copy changes around it while still proving the state was reached.
• State - assert a named control is disabled or checked, which catches logic regressions that leave the page looking correct.
• Negative - assert no error message is visible, the assertion most often missing from flows that pass while quietly erroring.
• Comparative - assert a result count is greater than zero, useful where the exact figure is data-dependent and only the threshold is meaningful.

One phrasing trap is worth calling out because it fails silently. Asking an agent to "tell me the price" is observational, and the value may appear in a summary without ever being captured for a later step to use. Naming the variable explicitly, as in store the price as a named value, is what persists it into the run output where CI can parse it.

Where the intent then lives is a separate decision from how it is phrased, and it decides whether the suite is reviewable. A one-off objective typed at a terminal is fine for exploration and leaves nothing behind. Test.md, the markdown-based framework in Kane CLI, persists the same plain-English objectives as step-level headings in a file, with front matter for environment variables and an import mechanism for shared flows like login. That form is worth reaching for as soon as a test matters, because a markdown file is reviewable in a pull request and a typed command is not.

What Should Stay Script-Based

Every vendor page on this topic argues for converting everything. That is the wrong shape of advice, because some tests get worse under semantic resolution. Use this rule: convert the test when the outcome is the point, and keep the script when the path is the point.

The step-count limit is a genuine constraint rather than a caveat to wave past. An agent's reasoning loop drifts as an objective grows, and past roughly fifteen steps reliability starts dropping. Splitting a long journey into shorter objectives that each carry their own assertions is the fix, and it also makes a failure point to a smaller region of the flow.

Tests that fail intermittently deserve their own decision. Converting a flaky selector-bound test to intent may remove the flakiness if the cause was selector churn, and will not touch it if the cause is a race condition or test data. Diagnose before converting, because an intent-based test that inherits a real race will still be flaky and will now be harder to read. Our guide to flaky tests covers separating those causes.

Migrating an Existing Suite Without a Rewrite

Nobody converts a mature suite in one pass, and nothing about this approach requires it. A practical sequence that keeps the existing suite authoritative throughout:

• Rank by maintenance cost - Pull the tests that have been edited most often in the last two quarters. Selector churn concentrates, and those files are where conversion pays back first.
• Restate one test as an outcome - Take a single high-churn journey and write what it must prove, including the negative assertion that nothing errored. Keep the original test running.
• Run both against the same build - Agreement on a healthy build tells you the intent is specified correctly. Disagreement is informative either way, and often exposes that the old test was asserting less than you assumed.
• Force a failure on purpose - Break the behavior deliberately and confirm the intent-based test goes red. An intent that has never failed is not yet a test, and this is the step that catches a false pass before it is load-bearing.
• Retire the script, keep the export - Only after the intent version has failed correctly at least once. Generated tests export to Selenium, Playwright, Cypress, and Appium, so the framework code remains available if you want it as the system of record.

Step four is the one teams skip and the one that matters most. Every other step tells you the test can pass. Only a deliberate break tells you it can fail, which is the only property that makes a green run mean anything. For terminal-driven and CI-driven runs the same objectives execute through Kane CLI, which reports per-run outcomes your pipeline can parse.

Conclusion

Start with the single test your team has repaired most often this quarter. Rewrite it as an outcome with at least one positive assertion and one negative assertion, run it alongside the original against the same build, then break the behavior on purpose and confirm it goes red. That one exercise teaches more about intent-based testing than any amount of reading, and it costs an afternoon.

The measurement in this article is the argument in miniature. A rename broke both selector-bound strategies and left the behavior untouched, while the locator bound to what the control was for kept resolving. Tests attached to purpose survive changes that tests attached to structure do not.

To author from intent and keep the framework code, KaneAI generates tests from natural-language prompts, requirement documents, and pull requests, re-anchors steps when the UI changes, and exports to Selenium, Playwright, Cypress, or Appium. The getting started with KaneAI documentation walks through the first test end to end. If you want to see how intent-shaped authoring reads before committing to it, our write-up on natural language test automation covers the authoring layer, and self-healing test automation covers what happens after a locator moves.