Author End-to-End Tests in Natural Language with KaneAI

Plan, Author, and Evolve E2E Tests for Web and Mobile Apps in Natural Language.

AutomationAI Testing

e2e (formerly TesterArmy): Open-Source Agentic Test Framework Explained

e2e is an open-source agentic test framework for web and mobile apps. See how its tests, record and replay, models, and CI work, and how to run it on TestMu AI.

Last Updated on:

Scripted end-to-end tests break when a selector or a flow changes, and fully agentic tests ask a model to rediscover every step on every run. The open-source e2e framework sits between the two: an agent handles the steps you describe in plain English, and ordinary locators and assertions check the result.

It is licensed under Apache-2.0, and its tests run locally, in CI, and on hosted browsers and devices, including TestMu AI's cloud Chrome, Edge, emulators, simulators, and real devices through the official @e2e-dev/testmu integration.

TesterArmy is now e2e, the same team under a new name. The hosted TesterArmy platform stays available until January 15, 2027, so teams on it can move their tests to e2e or compare e2e (formerly TesterArmy) alternatives.

TL;DR

e2e is an open-source, Apache-2.0 end-to-end testing framework for web and mobile apps in which a test step can be a plain-English goal that an agent completes, checked by locators and assertions in the same test. Verified agent steps replay from cache without model calls, and any AI SDK model can drive the agent.

  • Agent steps: Does every e2e test need an AI model? No. Tests built only from locators and assertions run without one; agent.act and agent.assert steps need an AI SDK model.
  • Record and replay: Do replayed agent steps call the model? No. After a later check verifies an agent.act step, e2e replays its recorded actions on the next run without a model call.
  • Mobile apps: Does e2e support mobile apps? Yes. The @e2e-dev/mobile engine runs the same tests on iOS simulators and Android emulators through agent-device, and hosted integrations add cloud devices, real devices included.
  • Migration: Can you migrate Playwright or Cypress tests to e2e? Yes. The e2e docs include migration guides from Playwright, Cypress, Selenium, Appium, Detox, Maestro, and TesterArmy.
  • Cloud runs: Can e2e tests run on TestMu AI? Yes, web and mobile. The official @e2e-dev/testmu package runs web targets in TestMu AI's hosted Chrome and Edge, one session per worker, and mobile targets on emulators, simulators, and real devices.

What Is the e2e Framework?

e2e is an open-source end-to-end testing framework for web and mobile apps, licensed under Apache-2.0. Its GitHub README describes the model: you describe a goal in natural language, an agent drives the app to reach it, and locators and assertions in the same test check the result. The same README marks e2e as in active development on the way to 1.0, so APIs and config can still change between minor releases.

Running npx e2e init scaffolds a project. The wizard asks for an engine and a model provider, including None for tests without AI, writes e2e.config.ts and an example test, and offers to set up your coding agent with the e2e skill so it can write and run tests.

Youtube thumbnail

How Does a TesterArmy e2e Test Work?

An e2e test combines fixtures: app opens the app, agent takes plain-English steps, and screen finds elements for exact assertions.

The agent steps docs define a step as one agent.act, agent.assert, agent.waitFor, or agent.extract call, and every step has a deadline and a model-call budget.

This test gives the agent a goal, then checks the outcome with a locator:

// tests/agent-form.e2e.ts
import { test, expect } from 'e2e';

test('an agent fills the form from a plain-English goal', async ({ app, agent, screen }) => {
  await app.open('/selenium-playground/simple-form-demo/');
  await agent.act('enter "Agent run on TestMu AI" in the single message field and submit it with Get Checked Value');
  await expect(screen.getByText('Agent run on TestMu AI', { exact: true })).toBeVisible();
});
  • agent.act - completes a goal such as filling and submitting a form. The model reads a text snapshot of the screen and asks for actions, which the runner performs and records.
  • agent.assert - a model judgment about what is on screen, which sees the instruction and the current screen but not the acting agent's summaries.
  • screen and expect - deterministic locators such as getByRole, getByText, and getByPlaceholder, with matchers such as toBeVisible, that check exact results without a model. The Playwright best practices guide recommends the same style of locator, with role locators first, since they find elements the way users do.

The snapshot the model reads leaves out raw HTML, cookies, headers, and environment values, masks password fields, and replaces configured secrets with a placeholder. A test with no agent steps never calls a model, so existing locator checks can sit in the same suite as agent steps.

How Does Record and Replay Cut Model Calls?

The replay cache records the actions in an agent.act() step after a later check verifies the result. On the next run, the runner replays those actions without calling the model, and if a control or the expected end state no longer matches, the agent takes over from the current screen.

The first run of the test above spends model calls while the agent works out the steps:

   ✓ an agent fills the form from a plain-English goal > agent.act "enter "Agent run on TestMu AI" ..." 16.20s · 3 model calls
 ✓  testmu-chrome  tests/agent-form.e2e.ts (1 test) 29.14s ai 28.4k tokens

 Test Files  1 passed (1)
      Tests  1 passed (1)
         AI  28.4k tokens · 3 model calls · google.generative-ai/gemini-2.5-flash
      Cache  1 missed

The second run replays the recorded actions with no model call:

   ✓ an agent fills the form from a plain-English goal > agent.act "enter "Agent run on TestMu AI" ..." 6.11s
 ✓  testmu-chrome  tests/agent-form.e2e.ts (1 test) 19.73s

 Test Files  1 passed (1)
      Tests  1 passed (1)
      Cache  1 replayed
  • Matching by meaning - replay finds each recorded control by its role, name, test id, and surrounding context, and its end check skips text that differs on every run, such as dates, durations, and ids.
  • Live judgments - agent.assert, agent.waitFor, and agent.extract always run live, so checks never come from a recording.
  • Run summary - each cached step is reported as replayed, handed off, or missed, and npx e2e run --no-cache runs every goal live, which shows the agent still reaches it without a recording.

Replay is what makes agent tests affordable on every pull request: model calls happen when the app changes. For a rollout plan built around steps like these, see the agentic E2E testing playbook.

Which Models, Engines, and Platforms Does e2e Support?

Agent steps use the model set in e2e.config.ts, and there is no default. The models docs cover a gateway such as Vercel AI Gateway or OpenRouter, a provider API key, or a local or self-hosted server, and the built-in agent needs a model that supports tool calls and images.

Plans you already pay for work too. The subscriptions guide signs in with npx e2e login openai for ChatGPT Plus or Pro, github-copilot for GitHub Copilot, opencode-console for OpenCode Console, and spacexai for SuperGrok or X Premium+. Claude Max and Team plans have no sign-in for other apps, so they connect through an Anthropic API key paid from the plan's monthly API credits.

Engines and add-ons ship as separate npm packages:

PackageWhat it does
e2eSDK, runner, and CLI
@e2e-dev/webBrowser engine built on Playwright: Chromium, Firefox, and WebKit
@e2e-dev/mobileiOS simulators and Android emulators, built on agent-device
@e2e-dev/testmuTestMu AI hosted Chrome and Edge for web targets, and Android emulators, iOS simulators, and real devices for mobile targets
@e2e-dev/kernel, @e2e-dev/lightpanda, and @e2e-dev/smolOther browsers for the web engine: Kernel's hosted Chromium, the Lightpanda headless browser, and smol machines Chromium microVMs on your own machine
@e2e-dev/eas and @e2e-dev/limrunHosted iOS simulators and Android emulators for the mobile engine
@e2e-dev/decisionDecision-model executors, plus a test filter that leaves out tests a pull request is unlikely to affect
@e2e-dev/githubPull request comment reporter for GitHub Actions

Device tests use the same agent, screen, and expect APIs as browser tests. The mobile docs require Xcode with an iOS simulator runtime or the Android SDK with an emulator for local runs, checked with npx e2e doctor. A hosted device provider removes that setup: each worker slot leases a cloud device when the run starts and releases it when the run ends.

The API testing docs show API tests written with fetch and schema matchers, in the same files and the same run as UI tests.

For existing suites, the e2e docs include migration guides from Playwright, Cypress, Selenium, Appium, Detox, Maestro, and TesterArmy.

How Do You Run e2e in CI?

Deterministic and agent tests run in the same CI job. In e2e's CI guide, each job installs the dependencies, runs npx e2e run --reporter list,junit, and uploads the .e2e/ folder, which holds report.json and junit.xml.

Browsers install through the web engine's own command, npx @e2e-dev/web install chromium --with-deps, which the web engine reference ties to the engine's pinned Playwright version.

  • Pull request comments - @e2e-dev/github turns each run into one comment on the pull request, edited in place on reruns, with the same text in the job summary.
  • Trace pages - each failed, timed-out, or flaky test gets a trace.md under .e2e/results/ with its steps, the app log, the agent's last turns, and a screenshot and semantic tree of the screen at failure.
  • Faster loops - npx e2e run --last-failed reruns what failed, and --shard 2/3 splits the suite across CI jobs.

Who Verifies the Agents Running Your Tests?

Once an AI agent is part of test execution, its own account of what it did cannot be the evidence. e2e builds that rule in: an agent.assert judgment does not see the acting agent's summaries, and only an agent.act step that a later check verifies is recorded for replay. The OpenAI Hugging Face incident shows why: some agents spoofed tool calls in their own transcripts, expecting a scorer to read them.

Agents your team builds need the same treatment once they act on real systems. TestMu AI Agent Assurance tests how your agents actually behave across workflows, tools, and actions: it invokes the agent for real and grades each criterion against observed evidence, such as files changed, artifacts produced, and tool calls, instead of the agent's summary of its work, and reports what it could not verify apart from the pass rate. It is self-serve with a 14-day free trial, the plans are on the Agent Assurance pricing page, and it runs from the terminal as rook.

For coding agents, Kane CLI runs a plain-English browser check against the running app from the terminal before you open a pull request, and returns a pass or fail backed by an evidence pack. In agent mode it emits NDJSON, so a coding agent or a CI job can read the verdict. Other agent-driven options are compared in the roundup of agentic AI testing tools.

Kane CLI - Testing Agent in Your Terminal

How Do You Run e2e on TestMu AI?

e2e ships an official TestMu AI package, @e2e-dev/testmu, documented in its TestMu AI integration guide. Tests stay the same: only the target's browser or device in e2e.config.ts changes, and both providers read LT_USERNAME and LT_ACCESS_KEY from the environment, so in CI they live as pipeline secrets.

For web targets, testmuBrowsers() gives each worker slot its own Chrome or Edge session on TestMu AI Automation Cloud, and every test attempt gets a fresh browser context in that session. It attaches over the CDP endpoint from the Puppeteer testing docs, the same remote-browser pattern as a Playwright grid:

  • Install - add @e2e-dev/testmu next to e2e and @e2e-dev/web, and export your TestMu AI username and access key.
  • Connect - set the web target's browser to testmuBrowsers(), with LT:Options fields such as video, network, and console passed directly in capabilities, as in the config below.
  • Run - run npx e2e run as usual, keeping --workers times the number of TestMu AI targets within your plan's parallel sessions. Extra sessions wait in TestMu AI's queue.
  • Review - each session appears in the TestMu AI dashboard as e2e-<run id>-<target>-<slot> under its build, with video, network, and console logs when the capabilities turn them on. Pass or fail stays in e2e's report.
// e2e.config.ts
import type { E2EConfig } from 'e2e';
import { web } from '@e2e-dev/web';
import { testmuBrowsers } from '@e2e-dev/testmu/web';

export default {
  targets: [
    {
      name: 'testmu-chrome',
      engine: web({
        browser: testmuBrowsers({
          capabilities: {
            browserVersion: 'latest',
            platform: 'Windows 11',
            build: 'e2e on TestMu AI',
            video: true,
            network: true,
            console: true,
          },
        }),
        viewport: null,
      }),
      app: { url: 'https://www.testmuai.com' },
    },
  ],
} satisfies E2EConfig;

The provider accepts Chrome and Edge only, so Firefox and WebKit targets keep running locally. The integration guide lists the other limits: a session must accept the connection within 60 seconds, a dropped connection fails the attempt, and browser.waitForDownload fails because downloads land on TestMu AI's machine. For an app on localhost or a private staging server, add tunnel and tunnelName to capabilities and start the tunnel as described in testing locally hosted pages.

For mobile targets, testmu() leases an Android emulator, an iOS simulator, or a device from Real Device Cloud for each worker slot and installs your build on it, so the machine running e2e needs no Mac, Xcode, or Android SDK. Virtual devices are the default, and deviceType: 'real' picks a physical phone:

// e2e.config.ts
import type { E2EConfig } from 'e2e';
import { mobile } from '@e2e-dev/mobile';
import { testmu } from '@e2e-dev/testmu';

const apk = 'https://prod-mobile-artefacts.lambdatest.com/assets/docs/proverbial_android.apk';

export default {
  targets: [
    {
      name: 'android-real',
      engine: mobile({
        platform: 'android',
        device: testmu({ device: 'Pixel 6', osVersion: '14', app: apk, deviceType: 'real' }),
      }),
      app: { bundleId: 'com.lambdatest.proverbial' },
    },
  ],
} satisfies E2EConfig;

The device and osVersion values must match TestMu AI's catalog exactly, and app takes an lt:// app id, an https URL, or a local path that e2e uploads. Per the integration guide, a device session is billed from the lease, which takes about 30 to 75 seconds, until the run releases it. Device settings, system alerts, and device logs through agent-device are not available, so read logs on the TestMu AI dashboard.

TesterArmy saved tests are plain-language steps that live on the platform. The TesterArmy migration guide has a coding agent rewrite them as e2e tests in your repository, or you can re-create them in KaneAI.

How Do You Get Started With e2e?

The quickstart asks for Node.js 24.8 or newer, or 22.22.3 or newer on Node.js 22, and for WSL on Windows.

Run npx e2e init, pick the web engine, and write one test that pairs an agent.act step with a locator assertion. Run it twice to see the replay, then add a TestMu AI target from above when you need parallel cloud browsers or real devices.

Conclusion

Start with the flow that breaks most often in your current suite: rewrite it in e2e and check whether the replay holds up over a few pull requests. Then move the rest of the suite across, and put the targets that need parallel browsers or real devices on TestMu AI. Teams on hosted TesterArmy should begin with the migration guide, so their saved tests move before the platform closes.

Testing web or mobile apps without writing code? KaneAI by TestMu AI turns plain-English instructions into tests.

To run your first test, start free, or
for a full walkthrough of the product.

Author

...

Srinivasan Sekar

Blogs: 21

  • Twitter
  • Linkedin

Srinivasan Sekar is Director of Engineering at TestMu AI, where he leads engineering and open-source initiatives behind the Selenium and Appium automation grid and owns TestMu AI's MCP Server. A committer to Appium and a contributor to Selenium, WebdriverIO, Taiko, and AppiumTestDistribution, he brings over 15 years of experience in quality engineering and open-source technologies. He is the author of the Apress book 'The MCP Standard: A Developer's Guide to Building Universal AI Tools with the Model Context Protocol,' a Certified Kubernetes and Cloud Native Associate, and an international conference speaker. Before TestMu AI he spent over eight years at Thoughtworks as a Principal Consultant and Quality Architect. Srinivasan holds a B.Tech in Information Technology from Anna University.

Reviewer

...

Mayank Bhola

Reviewer

  • Linkedin

Mayank Bhola is Co-Founder and Head of Products at TestMu AI, where he leads the entire product portfolio across KaneAI, Kane CLI, HyperExecute, SmartUI, the Real Device Cloud, Accessibility, and other software testing product lines. As an early Lead Architect he designed and built the company's flagship Tunnel technology from scratch, created the React-based automation platform, and architected the data-intensive pipelines and FAAS services that scale it. He brings more than 10 years of experience in software development and product engineering, with earlier roles as Head of Technology at Juggernaut Books and Senior Software Engineer at PressPlay TV and Zomato. Mayank holds a B.Tech in Computer Engineering from JIIT Noida.

Add to Google preferred sources

Summarise with AI

Copied to Clipboard!
...

3000+ Browsers. One Platform.

See exactly how your site performs everywhere.

Try it free
...

Write Tests in Plain English with KaneAI

Create, debug, and evolve tests using natural language.

Try for free

e2e Framework FAQs

Did you find this page helpful?

More Related Blogs

TestMu AI forEnterprise

Get access to solutions built on Enterprise
grade security, privacy, & compliance

  • Advanced access controls
  • Advanced data retention rules
  • Advanced Local Testing
  • Premium Support options
  • Early access to beta features
  • Private Slack Channel
  • Unlimited Manual Accessibility DevTools Tests