Run End-to-End Tests from Your Terminal with Kane CLI
Author and Run Tests Right from the Terminal in Natural Language, with an Evidence-Backed Pass or Fail from a Real Browser.
npm install -g @testmuai/kane-cli
Devin Bots vs Kane CLI: Checking Bot Pull Requests Before Merge
Devin Bots build and test their own pull requests on video. Kane CLI gives each PR an independent pass or fail, replays saved tests in CI and keeps evidence.
Published on:
When a coding agent can merge its own pull request, the check in CI is the last thing that looks at the change before your users do. Cognition launched Devin Bots on 17 November 2026, and its announcement describes a Bot merging pull requests that pass CI and Devin Review.
TL;DR
Devin Bots are Cognition's persistent coding agents, launched on 17 November 2026, that build features and test their own pull requests on video. Kane CLI by TestMu AI is a separate testing agent that replays saved plain-English tests in CI, checks every acceptance criterion and keeps evidence. Bots build the change; Kane CLI gates the merge.
- Best for delegating engineering goals: Devin Bots - a Bot keeps memory across sessions, splits a goal across parallel Devin sessions that each run in their own VM, and surfaces only the decisions that need a person.
- Best for an independent merge gate: Kane CLI - a verification step separate from the coding agent decides pass or fail against every acceptance criterion, and exit codes 0 to 3 let any CI runner block the merge, auto-approved pull requests included.
- QA Engineer template: The Devin Bots QA Engineer template tests each pull request in a real browser, posts the video and reports what passed and what failed. Devin's testing docs run the pull request video after CI is green, not as part of CI.
- Saved test replay: A saved Kane CLI test replays unchanged steps from cache with no LLM cost, so checking hundreds of Bot pull requests a week adds replays, and the model is called only to author a test or heal a step.
What Are Devin Bots?
Devin Bots are persistent engineering agents from Cognition. A Bot takes a goal, not a single task, breaks it into pieces, and runs several Devin sessions in parallel until pull requests are ready to merge.
According to the Devin Bots page and Cognition's announcement, as read on 18 November 2026:
- A Bot keeps memory and context across sessions.
- Every Devin session runs in its own VM.
- A Bot surfaces only the decisions that need a person, which you can approve in one click.
- Work can start from Slack, Linear or a schedule, and you can text a Bot, call it or add it to a group chat.
- Engineers set the success criteria, which systems a Bot can access, what it can change, and when spending or releases need approval.
- Launch templates include Software Factory, Android Engineer, Chief of Staff, Latency Optimizer, Product Engineer and QA Engineer.
Cognition reports large output numbers in its Introducing Devin Bots announcement. Its Android Bot produced an Android app in 44 days with 1,134 merged pull requests and 2,841 screenshot tests, and Cognition says it merges more than 2,000 PRs a week across its own teams.
These are company-reported figures. AlphaSignal's report on Devin Bots noted that Cognition has not published sample sizes, median results, defect rates or rollback rates.
How Do Devin Bots Test Pull Requests?
Devin Bots test pull requests with Devin's own browser testing. The QA Engineer template "tests each pull request in a real browser and posts the video", and "reports what passed and what failed on the pull request."
In Cognition's announcement examples, when a previously passing check fails, one Bot reruns it to rule out timing noise and opens a fix with before-and-after recordings, and another merges pull requests that pass CI and Devin Review. Devin's testing and recordings documentation (read on 18 November 2026) describes how that testing works:
- Plan - Devin writes a test plan aimed at "the single most important end-to-end flow" that proves the feature works.
- Run - it runs the plan in its desktop, clicks through the app in a browser, and records an annotated video.
- Timing - Devin runs this test "after CI is green and any review comments are addressed", so the recording comes after CI, not as part of it.
- Scope - the docs say: "If you need more exhaustive coverage, use your existing test suites and CI rather than visual recording."
In Cognition's May 2026 engineering post on verifying agentic development, Devin marks its own assertions as passed, failed or untested, and the QA Engineer template now reports what passed and what failed on the PR. The agent that wrote the change is the one grading it, and the testing docs do not describe an exit code or saved test that CI can enforce.
The same post is open about two limits: "a screenshot taken too early or too late can miss the toast entirely", and models may "lean too heavily on executing JavaScript in the browser" instead of clicking through like a user.
What Is Kane CLI?
Kane CLI is an end-to-end testing agent for developers and their coding agents, built by TestMu AI. You describe a user journey in plain English, and Kane CLI runs it in a real Chrome browser or on a mobile target and returns a pass or fail backed by evidence.
Any agent that can run a shell command can call it, Devin included, as well as GitHub Actions and any other CI runner. The details below come from the Kane CLI product page and docs, read on 18 November 2026.
npm install -g @testmuai/kane-cli
kane-cli login
kane-cli run --url https://staging.example.com \
"Sign up with a new email, choose the Pro plan, and
verify the first invoice shows the Pro price"
# From a coding agent or in CI:
kane-cli run "verify signup and first invoice on staging" \
--headless --agent- Exit codes - 0 for pass, 1 for fail, 2 for a setup or auth error, and 3 for a timeout or cancellation.
- Agent mode - the --agent flag returns structured NDJSON that any agent can read, one event per line. A walkthrough of that stream is in how a coding agent reads a Kane CLI run.
- CI sign-in - kane-cli login opens a browser for OAuth, so a CI runner passes a username and access key from the pipeline's secrets instead.
Here is one step of a real Kane CLI run: a plain-English check on an ecommerce playground demo store that adds an iPhone to the cart. The step screenshot carries the step result in its caption.

What Changes When Devin Bots Can Auto-Merge Pull Requests?
Devin Bots can merge on their own. Fortune's report on the launch says teams decide which changes need human review and can auto-approve low-risk edits, such as a button colour change.
When no person looks before the merge, the check in CI is the last line of defence, and it should not be graded by the agent that wrote the change. The general argument is in why AI coding agents need an independent tester.
Volume makes this sharper. A video per PR works when a person watches each one, but at hundreds of merged PRs a week nobody watches them all, while a CI gate that returns a machine-readable verdict scales with the PR count.
- A separate verdict - in Kane CLI the model picks the route, and a separate verification step decides the verdict. A pass means every acceptance criterion for the scenario was met, not that a model said the page looked fine.
- Healing never rewrites a pass - when the UI changes, step-heal re-finds an element by what it is, and flow-heal re-plans the route only as a last resort. Healing changes the path, never the acceptance criteria, and a wrong total is reported as a bug.
- A verdict CI can read - exit codes block the merge in any CI runner, with no parsing of prose.
How Does Kane CLI Keep Checks Cheap When Devin Bots Open Hundreds of PRs?
Kane CLI calls the model only when it has to. Each test moves through three states:
- Author - the model works out the path once. Each step snapshots the DOM, the accessibility tree and a screenshot, acts, then asserts, DOM first with vision as the fallback.
- Replay - the saved test (a _test.md file, committed with its cached recordings) replays unchanged steps from cache with no agent and no LLM cost. This is what runs in CI.
- Heal - the model comes back only when replay cannot find its target.
Replay skips the model, so a saved check costs far less to run than an agent re-exploring the app on every PR. At the weekly PR volume Cognition reports for its own teams, the cost that matters is model work per PR, and Kane CLI spends it only when a test is first written or needs to heal.
Devin Bots vs Kane CLI: Comparison Table
Devin facts were read on 17 and 18 November 2026 against the sources linked in this article; Kane CLI facts come from its product page and docs on 18 November 2026.
| Area | Devin Bots (Cognition) | Kane CLI (TestMu AI) |
|---|---|---|
| What it is | Persistent agents that run teams of Devin sessions toward a goal | Testing agent that verifies user journeys and gates releases |
| Main job | Building, fixing and shipping code | Proving a flow works before it merges |
| How PR testing runs | QA Engineer template: Devin writes a test plan per PR, clicks through in a browser and records a video | Plain-English objective; author once, replay from cache, heal only on change |
| Where it runs | Every session in its own Cognition VM | Local real Chrome, Android Emulator, iOS Simulator; TestMu AI cloud and real devices |
| CI replay without an LLM | Not described; the recording test runs after CI is green | Yes, a saved _test.md file replays unchanged steps with no LLM cost |
| Pass or fail verdict | Devin marks its own assertions passed, failed or untested and reports them on the PR with a video; no documented exit-code gate for that recorded check | Every acceptance criterion must pass; exit codes 0 to 3 |
| Who judges the result | Devin grades its own recorded browser check; Fortune reports Bot code also runs the team's existing tests | Kane CLI, separate from the coding agent |
| Who writes the lasting tests | The Bot can write coded tests with its own changes (Android Bot: 2,841 screenshot tests); the QA template's browser check is per PR | Plain-English tests written once from your acceptance criteria, saved in your repo, independent of the agent changing the code |
| Evidence and audit | Video and results on the PR or in the session | .evidence file with steps, screenshots, console and network logs and the verdict; opens at evidence.lambdatest.com, in Test Manager, or from the GitHub Action's PR comment |
| Mobile | Browser in a cloud VM; Android builds shipped to internal testing (Android Engineer template) | Android Emulator, iOS Simulator and real devices; 50,000+ real devices on TestMu AI |
| Pricing model | No separate Bot price as of 18 Nov 2026; Bots come with the Max, Teams and Enterprise plans, not Free or Pro | No charge to install; runs use KaneAI credits; new accounts can claim 10,000 credits valid for 30 days |
| Fits | Delegating ongoing engineering goals | An independent merge gate for code from any agent |
Where Are Devin Bots Stronger Than Kane CLI?
Devin Bots do things Kane CLI does not try to do:
- Building software - Kane CLI does not write features or fix bugs. Devin Bots do, across many parallel sessions.
- Owning a long-running goal - a Bot keeps context for days or weeks, such as keeping an Android app in step with iOS.
- Team workflow - Bots join Slack, take work from Linear and schedules, and escalate only the decisions that need a person.
- Built-in quick checks - the QA Engineer template gives each PR a recorded click-through without extra setup, a useful first look for a reviewer.
- Enterprise controls - the Devin Bots page lists SOC 2, ISO 27001, RBAC, SSO, SCIM, OIDC, audit logs and private networking, and Fortune reports that administrators control which repositories and tools each Bot can reach.
If the job is to get more engineering work done, Devin Bots are built for it. Kane CLI is built to decide whether each resulting change should ship.
Why Does Kane CLI Work as the Release Gate for Devin Bots?
Kane CLI works as the release gate because the agent that wrote the change does not grade it, and its saved tests replay the same steps on every PR with no model in the loop.
- Independent verdict - Kane CLI decides pass or fail, not the Bot that opened the PR.
- The same replay every time - with no LLM in the loop, a red build points to a change in the app or its environment, not a model taking a different route.
- Pass means every criterion - not the builder's own assertions, and not a video someone has to watch. A wrong total stays a bug.
- The suite Devin's docs point to - those docs say to use existing test suites and CI rather than visual recording for exhaustive coverage. Kane CLI gives you that suite in plain English, saved in your repo.
- Assertions that do not rest on one screenshot - Kane CLI acts through the UI and asserts each step against the DOM first, with vision as a fallback.
- Evidence an auditor can open - the .evidence file can be stepped through at evidence.lambdatest.com without rerunning the test.
- Real devices and many browsers - the same tests can run on the TestMu AI cloud through HyperExecute and the real device cloud, with 50,000+ real devices.
- Agent-agnostic - Kane CLI checks code from Devin, Cursor, Claude Code or a person the same way, so the gate does not change when your agent mix does.
How Do I Test Devin's Android Bot Builds?
On the Devin Bots page, the Android Engineer template "builds Android features in Kotlin and ships them to internal testing." Kane CLI can run the same plain-English checks against that build; locally, point it at the APK on an Android Emulator:
kane-cli run "Log in as the test user, open Orders, and
verify the latest order shows status Delivered" \
--target emulator --app ./app.apk- Local runs - local mobile runs need an Apple Silicon Mac with Android Studio.
- Cloud runs - without one, saved tests can run on the TestMu AI cloud, including real Android devices. Setup is in mobile automation in Kane CLI.
- Division of work - keep the Android Bot's own screenshot tests as unit-level checks, and use Kane CLI for the end-to-end journeys you will not ship without.
How Do I Use Devin Bots and Kane CLI Together?
The setup is "Devin Bots build, Kane CLI checks":
- Write the journeys you care about as Kane CLI tests - for example sign up, upgrade plan, check out and cancel. Commit the _test.md files and their cached recordings to the repo, so CI replays them instead of authoring them again.
- Tell the Bot to run Kane CLI before it opens a PR - Devin can run kane-cli run "<objective>" --agent, read the NDJSON result and fix failures first. Put that rule in the Bot's instructions, which a Bot reads at the start of every session, and store the TestMu AI username and access key as Bot secrets. Near the top of the repo's AGENTS.md, link the Kane CLI instructions file for coding agents: Devin's AGENTS.md documentation says Devin reads the file before it starts coding and includes only its first 16 KiB automatically.
- Gate the merge in CI - run the saved Kane CLI tests with --headless --agent on every PR and make that job a required check. The Kane CLI GitHub Action fails the check on any non-zero exit code, which blocks the merge, auto-approved PRs included, and posts the verdict as a PR comment next to the Bot's video. The workflow file is in how to connect Kane CLI to GitHub Actions.
- Review failures, not every PR - reviewers open the .evidence file or the PR comment only when a check fails.
- Keep Devin Review for code - Devin Review looks at the code, and Kane CLI checks the running product. They answer different questions.
For a feature-by-feature comparison with Devin itself, see Kane CLI vs Devin.
How Do Devin Bots and Kane CLI Compare on Cost?
As of 18 November 2026, Cognition has not published a separate price for Devin Bots. On the Devin pricing page, Bots come with the Max plan, the Teams plan (up to 100 users) and custom Enterprise plans; the Free and Pro plans do not list them.
- Usage allowance - each paid Devin plan has a usage allowance that refreshes daily and weekly.
- Bot limits - Devin's Bots usage limits documentation says a Bot has no separate usage limit on Max, while on Teams its creator and admins can set an on-demand cap in dollars.
- Per-Bot budgets - Fortune reports that each Bot can be given its own budget.
- Kane CLI - Kane CLI costs nothing to install. Test runs use KaneAI credits from a TestMu AI account, and new accounts can claim 10,000 credits valid for 30 days.
- Replay cost - saved tests replay unchanged steps without an LLM, so each extra PR adds a replay, not a new model session.
What Are Kane CLI's Limits?
- A button that keeps its label but does something else - Kane CLI is built to fail that step rather than heal it, so you get a red build to investigate instead of a quiet pass.
- No product code - Kane CLI does not write your product code. It is a testing agent, so pair it with a coding agent such as Devin Bots.
- Local mobile hardware - local Android and iOS runs need an Apple Silicon Mac; other machines run mobile tests on the TestMu AI cloud.
Install steps, sign-in options and the full command reference are in the Kane CLI introduction docs.
Testing from the terminal? Kane CLI by TestMu AI lets SDETs and developers plan, write, run and debug tests from the command line.
Author
Bhawana is a Community Evangelist at TestMu AI with over 3 years of experience creating technically accurate, strategy-driven content in software testing. She has authored 50+ blogs on test automation, cross-browser testing, mobile testing, and real device testing. She also serves as Product Marketing Manager for Kane CLI, the command-line tool that runs browser automation from the terminal using natural-language flows in a real Chrome browser. Bhawana is certified in KaneAI, Selenium, Appium, Playwright, and Cypress, reflecting her hands-on knowledge of modern automation practices. On LinkedIn, she is followed by 6000+ QA engineers, testers, AI automation testers, and tech leaders.
Reviewer
Shahzeb Hoda is the Associate Director of Marketing and a Community Contributor at TestMu AI, leading strategic initiatives in developer marketing, content, and community growth. With 10+ years of experience in quality engineering, software testing, automation testing, and e-learning, he has authored and reviewed 70+ technical articles on software testing and automation. Shahzeb holds an M.Tech in Computer Science from BIT, Mesra, and is certified in Selenium, Cypress, Playwright, Appium, and KaneAI. He brings deep expertise in CI/CD pipeline automation, cross-browser testing, AI-driven testing practices, and framework documentation. On LinkedIn, he is followed by 3,700+ engineers, developers, DevOps professionals, tech leaders, and enthusiasts.
Devin Bots vs Kane CLI FAQs
Did you find this page helpful?
More Related Blogs
TestMu AI forEnterprise
Get access to solutions built on Enterprise
grade security, privacy, & compliance
- Advanced access controls
- Advanced data retention rules
- Advanced Local Testing
- Premium Support options
- Early access to beta features
- Private Slack Channel
- Unlimited Manual Accessibility DevTools Tests


















