Agentic Testing in the End-to-End (E2E) Stack: Fixing Reliability Problems of Playwright MCP Agents

What Is Agentic Testing?

Agentic testing is end-to-end testing in which an AI agent receives a goal, such as "reply in a thread and confirm it appears in All Threads", and figures out the steps itself by observing the app, taking actions, and verifying the result. Traditional E2E tests follow a fixed script of clicks and assertions.

Slack summed up the difference well: scripted tests enforce journeys, while agents verify goals. One rule, the design principle Kane CLI is built on, makes agentic tests trustworthy enough for CI: reasoning owns the path, verification owns the verdict. The agent may take any route it likes, but whether the test passed must be decided against written criteria, never by the agent's own account of what it did.

Agentic testing is one branch of AI automation testing, and it also goes by agentic AI testing or agentic QA. It is different from AI agent testing, which checks how an AI agent itself behaves.

What Did Slack Find When It Tested Agents on End-to-End Testing (E2E) Flows?

Slack ran agents through Playwright MCP and the Playwright CLI, and had an agent generate Playwright tests for a simple flow and a complex one, with 20 runs per configuration.

They reported agentic runs costing about $15-30 each. Only about one in five runs took the same sequence of actions. Most CLI failures came from login, timing, and session problems rather than reasoning. They concluded that agentic testing belongs at the top of the testing pyramid, for analysis and debugging, not frequent CI runs.

Their numbers are right for how those runs were built. The conclusion changes once you change the build.

Why Is Agentic End-to-End Testing (E2E) So Expensive?

Agentic E2E tests are expensive because the model API is stateless: every turn re-sends the system prompt and the whole conversation, including every browser snapshot taken so far. Slack found most tokens were content the model had already seen. Cost grows with the number of turns and how fast context grows, not with how smart the model is.

The fix is architectural. Don't let your coding agent hold the browser loop.

IN-LOOP (what Slack measured)
Coding agent ──► browser action ──► snapshot back into coding agent's context
     ▲                                                     │
     └──────────── repeat 40-85 turns, context keeps growing ┘

DELEGATED (Kane CLI)
Coding agent ──► kane-cli run "<goal>" --agent --headless
                     │  (testing agent drives Chrome in its own process)
                     ▼
Coding agent ◄── one run_end JSON line: status, summary, final_state, test_url

The coding agent's context grows by one result line per test, not by dozens of snapshots. The browser work happens in a separate process, which also makes it easy to run many tests in parallel. In agent mode, Kane CLI streams NDJSON events and always ends with a single run_end line.

Playwright MCP vs Playwright CLI vs Kane CLI: Which of These AI Testing Tools Should Your Agent Use?

Use Playwright MCP when you want your coding agent to explore a page interactively in the same session. Use Playwright CLI for scripted, step-by-step browser commands. Use Kane CLI when the agent needs a verified pass or fail it can act on, at CI speed, with evidence.

The comparison is about how an agent uses each tool for testing. Playwright itself is excellent, and it's why Kane CLI exports to it. If you are choosing for AI test automation in CI, start with the "Who decides pass or fail" row.

Where Does Agentic Testing Fit in the Testing Pyramid?

Agentic testing belongs at three levels of the testing pyramid, not just the top. With replay and criteria-based verdicts, agents can verify changes while you code, gate pull requests, and explore at night.

• Unit and integration tests. Deterministic and fast. Agents write them; they don't run them.
• Agent self-check while coding. Your coding agent verifies the flow it just changed before it says "done".
• Replayed agentic E2E gate on every pull request. Proven _test.md flows, replay quickly, heal when the UI moves, and fail the build with a typed exit code.
• Nightly agentic exploration. Open-ended goals with bug detection on, to find what nobody wrote a test for.
• Production bug reproduction and flaky-test triage. Turn a ticket into a reproducible run with evidence attached.

Keep exported Playwright code for the flows you want fully deterministic forever. Everything else runs as intended.

If your team calls it the test automation pyramid, the split is the same: deterministic tests at the base, agentic checks above them.

Note: Kane CLI is free to install, and local runs are free on a TestMu AI account. Create your free account to try the local workflows below.

10 Kane CLI Workflows for Your Agentic Testing Stack

Every command below comes from the Kane CLI docs. For more recipes, including network, console, and Core Web Vitals checks, see the Kane CLI cookbook.

npm install -g @testmuai/kane-cli

# Install the skill for Claude Code, Codex CLI, and Gemini CLI in one go
npx @testmuai/kane-cli-skill

# Agents can't complete browser OAuth, so use basic auth
kane-cli login --username "$LT_USERNAME" --access-key "$LT_ACCESS_KEY"
kane-cli whoami

kane-cli run "Log in as {{email}}, open Billing, and verify the plan shows Enterprise" \
  --url https://staging.myapp.com \
  --variables-file ./.testmuai/variables/staging.json \
  --agent --headless --timeout 300 --max-steps 50 \
  2>/dev/null | tail -1 | jq '{status, one_liner, reason, test_url}'

kane-cli run --analyzer-only --agent \
  --cdp-endpoint "$CDP_URL" \
  --condition "The pricing table shows three plans" \
  --condition "The Pro plan shows a monthly price"

# One test
kane-cli testmd run tests/checkout_test.md --headless --agent

# The smoke suite, 4 workers, stop at the first failure
kane-cli testrun run --match 'tests/e2e/.*' --tags smoke \
  --parallel 4 --on-failure fail-fast --headless \
  --username "$LT_USERNAME" --access-key "$LT_ACCESS_KEY"

name: Agentic E2E gate
on: [pull_request]

jobs:
  kane:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - uses: browser-actions/setup-chrome@v1
      - run: npm install -g @testmuai/kane-cli
      - name: Run smoke suite
        env:
          LT_USERNAME: ${{ secrets.LT_USERNAME }}
          LT_ACCESS_KEY: ${{ secrets.LT_ACCESS_KEY }}
        run: |
          kane-cli testrun run --tags smoke --parallel 4 \
            --on-failure fail-fast --headless \
            --username "$LT_USERNAME" --access-key "$LT_ACCESS_KEY"
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: kane-evidence
          path: |
            .testmuai/evidence/*.evidence
            ~/.testmuai/kaneai/sessions/

kane-cli plugin install remote-execution

kane-cli testrun run --match 'tests/web/.*' --remote --parallel 4 \
  --username "$LT_USERNAME" --access-key "$LT_ACCESS_KEY"

kane-cli run "Explore account settings: change the display name, upload an avatar, \
  switch the language to French and back, and report anything broken" \
  --url https://staging.myapp.com \
  --bug-detection continue \
  --agent --headless --max-steps 80 --timeout 900

kane-cli run "Add two items to the cart, apply code SAVE10, remove one item, \
  and verify the discount is recalculated on the remaining item" \
  --url https://staging.myapp.com \
  --final-validation on --agent --headless --timeout 300

for i in 1 2 3 4 5; do
  kane-cli run "Search for 'quarterly report' and open the first result" \
    --url https://staging.myapp.com --agent --headless --timeout 180 \
    > run_$i.ndjson 2>/dev/null &
done
wait
for f in run_*.ndjson; do tail -1 "$f" | jq -r '.status'; done | sort | uniq -c

kane-cli run "Sign up with a new email and verify the welcome screen" \
  --url https://staging.myapp.com --code-export --code-language javascript \
  --agent --headless

# Or regenerate code from an existing test.md recording
kane-cli testmd export tests/signup_test.md --language javascript

kane-cli doctor --target emulator --install
kane-cli devices list --target emulator

kane-cli run "Log in, open Orders, and verify the latest order shows Delivered" \
  --target emulator --device-name "Pixel 7" --os-version 14 \
  --app ./app-debug.apk --agent

kane-cli run "Open the pricing page and store the Pro monthly price as 'pro_price'" \
  --url https://myapp.com --agent --headless 2>/dev/null \
  | tail -1 | jq -r '.final_state.pro_price'

How Do You Keep Agentic Test Costs Under Control?

Most of the spend in AI E2E testing comes from re-sent context and repeated reasoning.

• Delegate the loop. The coding agent should read results, not snapshots.
• Replay what has passed. Author once with run, keep it as _test.md, replay with testmd run.
• Cap every run. Always set --max-steps and --timeout.
• Measure. run_end includes token_usage; track it per test like a performance budget.
• Fan out, don't chain. Run independent goals as parallel processes.
• Save exploration for nightly. Open-ended goals cost the most; run them on a schedule, not every commit.

How Do You Write a Good Agentic Test Objective?

A good objective states the goal and the outcome to verify. A bad one lists clicks or leaves the outcome vague.

Use {{variables}} for test data and secrets. Never paste credentials into the objective.

How Do You Verify These Claims on Your Own App?

Run Slack's experiment yourself. Pick one real flow and run it 20 times as a fresh agentic run, then 20 times as a replay. Compare pass rate, duration, and tokens. The numbers settle the argument for your app, not ours or Slack's.

The same comparison is a fair way to judge any AI QA testing claim, including the ones in this guide.

# 1. Twenty fresh agentic runs
OBJ="Create a channel, post a message, reply in a thread, and verify the reply appears in All Threads"
for i in $(seq 1 20); do
  kane-cli run "$OBJ" --url https://staging.myapp.com \
    --agent --headless --timeout 600 --max-steps 60 2>/dev/null \
    | tail -1 | jq -c '{status, duration, token_usage}' >> fresh.ndjson
done
jq -s '{runs: length, passed: (map(select(.status=="passed")) | length), avg_seconds: (map(.duration) | add / length)}' fresh.ndjson

# 2. Save the flow as tests/thread_reply_test.md, then twenty replays
for i in $(seq 1 20); do
  start=$(date +%s)
  kane-cli testmd run tests/thread_reply_test.md --agent --headless > /dev/null 2>&1
  echo "{\"exit\": $?, \"seconds\": $(( $(date +%s) - start ))}" >> replay.ndjson
done
jq -s '{runs: length, passed: (map(select(.exit==0)) | length), avg_seconds: (map(.seconds) | add / length)}' replay.ndjson

See Running test.md files for how to save a flow as a _test.md file.

Objections, Answered

"Isn't delegating the loop just moving the tokens somewhere else?" The first authoring run still uses a model; that cost doesn't disappear. Two things change. The coding agent's context stops growing with browser snapshots, which Slack identified as the main cost driver. And replays of steps that have already passed don't need fresh reasoning. Every run_end reports token_usage, so you can check this on your own flows.

"The verifier is AI too. Why trust its verdict?" You write the acceptance criteria, not the model. By default, assertions read the page DOM (--assertion-mode dom) and use vision only as a fallback. Every verdict ships with an .evidence pack anyone can open and replay, so a wrong verdict is visible and checkable.

"What about apps that really are nondeterministic?" Two of the usual causes of flaky tests are gone by construction: there are no hand-written locators to break and no hand-tuned waits to guess wrong. For real product nondeterminism, bug detection classifies what it sees and reports a confidence signal. Kane's own docs call this partial, and it is: no tool fully solves nondeterministic app state.

"Am I locked in?" No. _test.md files live in your repo as markdown, and any flow exports to Python or JavaScript Playwright with --code-export or kane-cli testmd export. You can leave with working code.

"Does my coding agent need to change?" No. The skill is a plain Markdown file for Claude Code, Codex CLI and Gemini CLI, and any agent that can run a shell command can call kane-cli run --agent and read the last line. For agent-specific setup, see the guides on testing apps built with Claude Code and testing apps built with Cursor.

"Is it safe with real credentials?" Pass secrets as variables marked "secret": true or from your CI secret store, never inside the objective. Local runs use your own Chrome on your machine.

What Doesn't Kane CLI Do?

• Local mobile runs need macOS on Apple Silicon. Other machines can run mobile suites on the cloud grid with testrun run --remote.
• One objective per run invocation. Parallelism comes from separate processes or testrun run --parallel N.
• No mid-run questions in CI. ask_user is turned off when there's no terminal, so flows that need a human for OTPs or CAPTCHAs should use test-mode bypasses in CI.
• Cloud runs need a plan with HyperExecute macOS runners.
• It needs a TestMu AI account. The free tier is enough to try the local workflows above.

When Should You Still Use Playwright MCP Directly?

Use Playwright MCP directly for interactive work inside one editor session: poking at a page, inspecting the DOM or building a selector by hand. For anything you want to repeat, gate a build on or hand to a teammate, use a testing agent that returns a verdict and evidence.

For MCP testing that should end in a verified pass or fail rather than raw MCP primitives, see how Kane CLI works as a layer above browser automation MCP.

Note: Install Kane CLI and claim 10,000 free credits.