Run End-to-End Tests from Your Terminal with Kane CLI
Author and Run Tests Right from the Terminal in Natural Language, with an Evidence-Backed Pass or Fail from a Real Browser.
npm install -g @testmuai/kane-cli
We Are All Verification Engineers Now
Shopify's CEO says we are all verification engineers now. What the role means when agents write the code, the four rules, and the tools.
Published on:
Agents can now write more code in a day than your team can review in a week. Shopify's CEO has named the job that's left.
At Vercel Ship in San Francisco, Tobi Lütke told Guillermo Rauch, "We are now all verification engineers. We are building harnesses, we are building acceptance tests, and we are building... verifiables." That's the line as quoted from the X session by @DarlingtonDev.
Tobi Lütke (@tobi), Shopify CEO, at Vercel Ship SF: "we are now all verification engineers. We are building harnesses, we are building acceptance tests, and we are building… verifiables."
- Mike Darlington ▲ (@DarlingtonDev) November 16, 2026
When agents write the code, the work moves to the checks:
→ harnesses that make agent output repeatable
→ acceptance tests that define "done"
→ anything a machine can verify without a dev squinting at a diff
Full fireside with @rauchg in the reply 👇
Vercel's own recap says it more plainly: "We now think of ourselves as verification engineers, building harnesses and acceptance tests that make changes verifiable."
Watch the full fireside with Tobi Lütke and Guillermo Rauch below.
That's the clearest description of the agent era. Writing the code became cheap. Proving it works didn't.
The engineering teams that are most needed now are the ones who can prove it's right at the speed that agents ship.
TestMu AI exists to do that job for them.
What Is Verification Engineering?
Verification engineering is the practice of defining what "done" means prior to code being written, then proving that every change satisfies that definition via repeatable, machine-checked tests and evidence. The verification engineer owns that work: the harness, the acceptance tests and the record of every run. When AI agents are writing the code, that's the core engineering job.
Why the Work Moved to the Checks
The numbers are from the same stage. In his keynote, Rauch shared Vercel's data for the last 12 months: developers on Vercel are shipping twice as many new projects, and their agent-triggered deployments are up 28x.
Agents now write, deploy and redeploy faster than any person can read a diff. The human job therefore moves on. You cease to write every line.
You start to own three things:
- What 'done' means. Written down prior to the build in terms a machine can check.
- How 'done' is proven. A run that either passes or fails in exactly the same way every time, with no one looking at a PR.
- What evidence is left behind. So anyone can see what was checked, when, and why it passed.
That's verification engineering. Testing used to be a phase at the end. It's now the primary control that a team has over what agents get shipped.
The industry is starting to say so. In July, The New Stack had an article about an 'automated verification engineer' role, a role which is 'completely focused on automating both testing and verification'.
In October, Paul Nashawaty from theCUBE Research wrote that teams who have invested in coding agents but not in verification are in fact producing output at a rate faster than they can verify it.
What Tobi Means by Harnesses, Acceptance Tests and Verifiables
Shopify has given a concrete example. As per Vercel's recap, Shopify is rewriting their storefront renderer from Ruby to Rust and will be doing a byte-for-byte comparison of the output prior to the switch.
That's verification engineering in one sentence. The old renderer defines what correct looks like.
The new one has to match it exactly. A machine makes the decision.
Take Tobi's three words one at a time:
- Harnesses make every run repeatable. The same input gives you the same run, so a pass today still means something tomorrow.
- Acceptance tests define 'done'. They start from the business outcome (the order total is right), not from the code.
- Verifiables are things that a machine can verify without anyone having to look at a diff: an exit code, a matching output, an evidence file.
Get those three right and agents can go as fast as they like. Get them wrong and speed just ships bugs faster.
The Trap: An Agent Grading Its Own Homework
Most teams completely overlook this bit.
If the same agent writes the code and the acceptance test, then it's effectively marking its own homework. A misread requirement ends up in both the fix and the test.
The test goes green. The feature is still wrong.
So verification engineering has four rules:
- People define 'done' prior to the build. It's the product that owns what 'done' means, not the agent and not the diff.
- The agent that wrote the code never grades it. The check runs outside of the loop that made the change.
- The verdict is an exit code and evidence. Pass or fail comes from the run, not from the model saying "all tests passed".
- The final check runs after the final change. The approver sees the verdict on the exact code that ships.
Break any one of these, and you don't have verification. You have a status report.
Verification Engineer vs QA Engineer vs SDET
The three roles have the same skills; just each one starts from a different question.
| QA engineer | SDET | Verification engineer | |
|---|---|---|---|
| Core question | Does the release work for users? | Can we automate the checks? | Can we prove every change meets "done", whoever wrote it? |
| When the work happens | Mostly after the build | During the build | Before the build, and after every change |
| Whose code they check | Code written by people | Code written by people | Code written by people and AI agents |
| Main output | Test cycles and bug reports | Automation frameworks and CI tests | Acceptance criteria, harnesses, pass/fail exit codes and evidence |
| Typical tools | Test cases, manual and automated suites | Selenium, Playwright, Cypress, Appium | Plain-English acceptance tests, cached replay, evidence records, CI gates |
Verification engineering is built on QA and SDET skills. It brings in ownership of 'done' and treats agent-written code in exactly the same way as any other code, i.e., it's checked from the outside with proof.
TestMu AI Is the Verification Layer
TestMu AI is the independent verification layer for software built by both people and agents. There's one rule under every product: no model ever verifies its own work, and every verdict is backed by evidence.
Reasoning owns the path. Verification owns the verdict. The AI gets through your app, and the model never actually votes pass or fail.
Here's how the platform maps to Tobi's three words.
Acceptance Tests: KaneAI and Kane CLI
KaneAI is TestMu AI's end-to-end testing agent for web and mobile. You describe the test in natural language (no scripts, no brittle selectors), and KaneAI will plan, write, and run it. Done is defined in the language of the business before any agent touches a line of code.
Kane CLI brings the same tests to developers and coding agents in the terminal. It works with Claude Code, Cursor, Codex, and Gemini CLI and can be used in GitHub Actions or any CI pipeline. There's an --agent flag that returns structured output that the coding agent can parse.
- Healing fixes the path; it doesn't fix the acceptance criteria. If a button has moved, then the Kane CLI will find it again. If the order total is wrong, then that's a bug, and it's reported as such and never healed into a pass.
- Every run ends in an exit code: 0 for pass, 1 for fail, 2 for setup or auth errors, 3 for timeout or cancellation. Your CI, hooks and agents gate on it.
Harnesses: Cached Replay and HyperExecute
A harness is only useful if a pass means the same thing every time. Save a Kane CLI run as a test, and its unchanged steps replay from cache with no LLM cost. Same app, same verdict, run after run.
Any step that had to be re-authored is recorded for review.
HyperExecute keeps the harness fast at scale. It runs your existing Selenium, Playwright, Cypress and Appium suites in parallel on just-in-time VMs, all configured from a single YAML file. Customers are seeing runs up to 70% faster.
Verification keeps up with the agent speed of their deploys rather than becoming the new bottleneck.
Verifiables: Evidence for Every Run
Every Kane CLI run saves its evidence (every stage with its screenshots, the console and network logs and the verdict). Runs upload to the TestMu AI Test Manager by default so the entire team sees the same record.
The person approving a release doesn't look at a diff. They look at proof.
When the Software Is an Agent: Agent Testing and Agent Assurance
More of what the teams ship is in itself an agent, and it too needs a verdict.
- Agent Testing tests chatbots, voice agents, phone agents and IVR systems the way a real user would. AI evaluators have real conversations with your agent, and a simulated caller then dials the number.
- Agent Assurance grades what the AI agent actually did (i.e., what it changed) rather than what it claims to have done. Its Rook CLI runs your agent against a suite of tests derived from your code and your docs and then checks the impact on tools, files and APIs. The result is one of three: pass, fail or unable to verify. Anything that it is unable to verify is simply not included in the pass rate at all.
What to Do This Quarter
Every engineering team now needs a verification plan as clear as its build plan. Start here:
- Pick your 5 most important user flows. Checkout, sign up, payment, search, account changes.
- Write done for each one in plain English. Product owns it, and it's written before the next agent touches that code.
- Put an independent check in CI. It gates on an exit code and runs outside of the agent that made the change.
- Keep the evidence for every run. A release decision should be based on the record, not on memory.
- Make verification a named role. Someone has to own the harness, the acceptance tests and the evidence. That person is your verification engineer.
The Job That Matters Now
Code is no longer the scarce skill. Agents write it, and they write a hell of a lot of it. The scarce skill is the skill of being able to say with proof that what's shipped is right.
Every engineer is now a verification engineer. TestMu AI gives them the capability to do it: acceptance tests in plain language, replay that gives the same verdict every time, evidence for every run, and agents checked by something other than themselves.
Agents find the path. Verification has the verdict.
Testing from the terminal? Kane CLI by TestMu AI lets SDETs and developers plan, write, run and debug tests from the command line.
Author
Asad Khan is the Co-Founder and CEO of TestMu AI, with over 15 years of experience in software testing, product strategy, and leadership. Under his leadership, TestMu AI has grown into an AI-native software testing platform trusted by 2.5M+ users across 132 countries, including startups, enterprises, and Fortune 500 companies. He is a member of the Forbes Technology Council and is followed by 39,000+ professionals on LinkedIn from the software testing and quality assurance community. Before TestMu AI, Asad co-founded 360logica, a software testing company that he grew into a multi-million-dollar business before its acquisition by Saksoft. Earlier, he began his career at GlobalLogic as a Lead Engineer serving top Wall Street banks. He holds a Bachelor of Technology from JSS Academy of Technical Education, Noida.
Reviewer
Maneesh Sharma is Chief Operating Officer at TestMu AI, where he leads the company's revenue functions across go-to-market, sales, RevOps, customer success, and the partner ecosystem, driving its global expansion. He brings over 28 years of experience across product, go-to-market, and business operations. Before TestMu AI he was Managing Director and General Manager for GitHub across India and APAC, Head of Partner Sales and Ecosystem at Adobe, Chief Revenue Officer at WizIQ, and Head of Products and Solutions at SAP India. Maneesh holds an MBA from IIM Bangalore and a B.E. in Electronics and Communications from NIT Surat.
FAQs
Did you find this page helpful?
More Related Blogs
TestMu AI forEnterprise
Get access to solutions built on Enterprise
grade security, privacy, & compliance
- Advanced access controls
- Advanced data retention rules
- Advanced Local Testing
- Premium Support options
- Early access to beta features
- Private Slack Channel
- Unlimited Manual Accessibility DevTools Tests

















