Why AI Agents Fail at End-to-End (E2E) Testing: 12 Limits and the Fix

1. The Test Can't Get Past Login

What you see: the agent writes a beautiful checkout test that never reaches checkout. It times out on the sign-in page, or burns tokens hunting for the auth file and base URL every session.

Why: authentication is shared state across the whole suite, and agents reason one file at a time. A developer who built an MCP server to patch this in Playwright MCP named login as problem number one.

Fix: auth belongs to the test run, set once, not reinvented inside every spec.

2. Every Suite Fakes the World Differently

What you see: one suite seeds fake auth data, another logs in for real, a third mocks the routes its own way. Each is internally consistent; together they contradict each other.

Why: an LLM optimizes the file in front of it. Nothing in the loop enforces one testing strategy across the app.

Fix: fewer fakes. A black-box test that drives the real app through the real UI has nothing to disagree with.

3. Mocks That Make the Test Pass and the Product Fail

What you see: shared route mocks fix one failure and cause another. The suite goes green while production breaks, because the test exercised your fake API, not the real one.

Why: the agent is rewarded for green, and a mock is the cheapest way to get green.

Fix: test the outcome the user sees on a real browser against a real environment. Mock only what you truly can't reach.

4. The Test Describes the Code, Not the Requirement

What you see: the agent writes an assertion that corresponds to what the code currently does, including the bug. As one Cursor forum member put it, it is "obsessed with writing tests to match the code."

Why: the code is in context, and the requirement usually isn't. That original poster's own fix was a rule: "There is passing, and there is passing CORRECTLY."

Fix: the requirement must be the test. Acceptance criteria written in plain language, by a human, separate from the implementation.

5. Selectors That Rot on the Next Refactor

What you see: a renamed class, a regenerated id or a moved button turns the build red. Nothing is broken for the user.

Why: a generated Playwright spec freezes today's DOM into getByTestId and CSS selectors. Playwright MCP also reads the accessibility tree, so on HTML that isn't clean and semantic, its locators often fail outright.

Fix: find elements by what they are and what they do, not where they sat yesterday.

6. Hand-Tuned Waits That Guess Wrong

What you see: waitForTimeout(3000) sprinkled through the spec. It passes on a fast laptop and flakes in CI.

Why: agents write non-idiomatic Playwright, reaching for manual timeouts instead of the framework's auto-waiting. That's a common source of flaky tests.

Fix: no hand-written waits at all. The runner waits for the state the step needs.

7. Green Alone, Red Together

What you see: each suite passes in isolation. Run them together, and they fail. This is the exact symptom that made the forum poster shelve his suite.

Why: leaked state, order dependence, and shared test data. These are whole-suite properties no single generation step can see.

Fix: independent tests with their own setup, run in parallel so ordering bugs surface on day one, not month three.

8. Paying the Model on Every Single Run

What you see: an "agentic" test that re-reasons the whole flow each time. It's slower than a script, costs tokens per run, and can take a different path on Tuesday than on Monday.

Why: if the model drives every run, every run is probabilistic. Slack Engineering measured about 40 turns and 3.8M tokens for an MCP-driven flow, and $15-30 per agentic run.

Fix: pay for intelligence once, then replay deterministically. (Section 4 shows how.)

9. Happy-Path Tunnel Vision

What you see: tests for valid card numbers, none for the double charge on a payment timeout, the second user editing the same record, or the permission edge case.

Why: the obvious path is the most likely completion. Edge cases arise from someone asking, "What could go wrong for the business here?"

Fix: a human states the risky outcomes as intents. An agent hunts for bugs beyond the stated path and reports them separately, without silently widening the test.

10. A Red Build That Tells You Nothing

What you see: CI fails. Is checkout broken, or did a cookie banner shift a button? Someone spends an hour finding out, every time.

Why: a script can only say "step 14 failed". It can't distinguish between a broken product and a moved element.

Fix: a verdict with a reason and a category, so red means the product broke, not that something moved.

11. "Trust Me, It Passed"

What you see: the agent reports success. You have its summary, maybe a screenshot, and no way to reconstruct what it actually did.

Why: chat output is not a record. For regulated teams especially, a verdict without evidence is AI-generated confidence.

Fix: every run emits a self-contained, tamper-evident record: steps, screenshots, network, console.

12. The Agent "Fixes" the Test Instead of the Bug

What you see: a test fails, you ask the agent to make CI green, and it loosens the assertion. Now the test checks something else.

Why: to an agent with write access, the test is just more code. Nothing separates the path from the pass criteria.

Fix: healing may change how the test gets there, never what counts as passing. A wrong total should be reported as a bug, not healed into green.

How That Maps to the 12 Failures

1. Install the CLI and the Agent Skill

npm install -g @testmuai/kane-cli
npx @testmuai/kane-cli-skill   # teaches Claude Code, Codex or Gemini CLI how to call Kane

2. Authenticate Non-Interactively

Agents have no TTY, so log in with flags rather than a prompt:

kane-cli login --username "$LT_USERNAME" --access-key "$LT_ACCESS_KEY"

3. State an Outcome, Not a Script

kane-cli run "Sign in as a returning customer, add the blue hoodie in size M to the cart, \
  check out with the saved card, and confirm the order total equals the cart total" \
  --url https://staging.yourapp.com \
  --agent --headless --timeout 300 --max-steps 40 \
  --final-validation on

--agent switches output to NDJSON, so the coding agent reads machine output instead of scraping terminal text. --final-validation on adds a final check of the end state against your stated outcome.

4. Let the Agent Read the Verdict, Not Guess It

The process exits 0 on pass, 1 on fail, 2 on a setup error, and 3 on a timeout, so CI and agents can branch without parsing prose. The last NDJSON line is the run_end event, which carries:

• status and one_liner: the verdict in a sentence
• reason: why it failed, in terms of your outcome
• final_state: where the browser ended up
• test_url and run_dir: where the evidence lives
• token_usage: what the run cost

That closes the loop the forum poster couldn't: the agent changes code, runs Kane, reads reason, fixes, and reruns until it exits 0. It never rewrites the acceptance criteria to get there.

5. Add One Rule to Your Agent Instructions

Paste this into CLAUDE.md, AGENTS.md or .cursor/rules:

## End-to-end verification
- Never write Playwright or Cypress specs by hand for user flows.
- After any change that touches a user-facing flow, run:
  kane-cli run "<the outcome in plain English>" --url $STAGING_URL --agent --headless
- Treat exit code 1 as a product bug. Fix the code, never the outcome sentence.
- Treat exit code 2 or 3 as environment problems and report them; do not retry blindly.

6. Promote What Matters Into a Committed Suite

Keep important flows as markdown test files in the repo, so the outcome sentences get reviewed in PRs like any other code:

kane-cli testmd run tests/checkout_test.md
kane-cli testrun run --tags smoke --parallel 4 --on-failure fail-fast

Need scale? kane-cli plugin install remote-execution, then kane-cli testrun run --remote --parallel 20 fans out on HyperExecute (this needs a HyperExecute macOS runners plan).

7. Hunt Bugs Beyond the Stated Path

kane-cli run "Complete checkout with a saved card" --url $STAGING_URL \
  --agent --bug-detection continue

The run still answers your outcome. Issues it notices along the way are reported separately, with a category and a confidence signal, rather than quietly expanding what the test checks.

8. Keep an Exit to Plain Playwright

If a team wants a script for a particular flow, export it. You're never locked into a format you can't read:

kane-cli run "..." --url $STAGING_URL --code-export --code-language javascript
kane-cli testmd export tests/checkout_test.md --language javascript

9. Audit Any Verdict

Every run writes a sha256-hashed .evidence pack. Drop it into evidence.lambdatest.com to replay the run step by step, with screenshots, network logs, console logs, and terminal output. On a failure, you also get the bug Kane reported, its category, and the verdict behind it. If you can't get from a verdict back to what produced it, that's a bug in Kane, not a limitation you should accept.

10. Mobile, Too

The same outcome-first approach runs against native apps on a local Android emulator or iOS Simulator (currently macOS on Apple Silicon):

kane-cli doctor --target emulator --install
kane-cli run "Log in and open order history" --target emulator \
  --device-name "Pixel 7" --os-version 14 --app ./app.apk

What Kane CLI Does Not Solve (On Purpose, or Not Yet)

A testing tool that claims to fix everything is selling something. Here is where the boundary is.

• Semantic drift is harder than moved elements. Healing is strongest when an element's address changes and its meaning doesn't. If a button keeps its label but now routes somewhere else, Kane would rather fail loudly than adapt. That's a deliberate choice, not a solved problem.
• Flaky software vs flaky tests is only partly solved. Kane removes two of the usual three causes, rotting locators and hand-tuned waits, by construction. For genuinely nondeterministic app state, the bug-detection pass gives a verdict with a confidence signal, not certainty.
• It won't pick your risks for you. Deciding that a double charge on a payment timeout matters more than an error is a product judgment. Kane makes each stated intent cheap to verify forever. The bottleneck becomes how many intents a human is willing to state, which is the part that should stay human.
• It isn't observability. Memory leaks, slow queries, and misconfigured infrastructure don't show up in a UI flow. Keep your monitoring.
• It doesn't replace your governance process. It does give that process something it rarely has: a tamper-evident record of what was tested, how it was tested, and what happened.