Run End-to-End Tests from Your Terminal with Kane CLI
Author and Run Tests Right from the Terminal in Natural Language, with an Evidence-Backed Pass or Fail from a Real Browser.
What Claude Code's Own Tests Catch, and What They Miss (2026)
Claude Code wrote 57 tests for a 12-requirement app and all passed. 12 Kane CLI checks agreed, its suite caught 8 of 8 seeded bugs, and a phone bug got past.
Published on:
A coding agent's test suite is only worth as much as the bugs it can fail on, and experienced engineers know it: in Stack Overflow's 2025 Developer Survey, developers with 10 or more years of experience had the highest "highly distrust" rate for AI output, at 20%.
So I gave Claude Code one written spec, asked it to build the app and test its own work, and then checked the same build three other ways: 12 Kane CLI objectives written before the build, 8 seeded bugs, and its own suite re-run at a phone screen width. Every count is below, including the requirements where its unit tests were all you needed.
Overview
Claude Code's own tests caught every defect inside the spec in this run: its 57 tests passed, 12 independent Kane CLI checks by TestMu AI agreed, and its suite failed on all 8 seeded bugs. What it missed was what the spec never stated: at phone widths the cart's Remove button sat off-screen, and no test looked there.
What Did Claude Code's Tests Catch and Miss?
- Requirement coverage: Claude Code's suite covered all 12 requirements, and the 12 Kane CLI objectives written from the spec before the build passed 12 of 12 against the same build.
- Seeded bugs: Claude Code's 57 tests failed on each of 8 one-line bugs applied to the finished app, so a green run from this suite carried real weight.
- Unit tests alone: Claude Code's unit and API tests caught 4 of the 8 seeded bugs, including both pricing-threshold bugs that every Playwright test missed.
- Browser tests alone: Claude Code's Playwright tests caught 6 of the 8 seeded bugs, including four page-state bugs that no unit test could see.
- Phone-width layout: Claude Code's browser tests missed a phone-width bug: below about 520 pixels wide the cart pushed the Remove buttons off-screen, and every test ran in a desktop window.
- Spec gaps: Claude Code filled unstated rules on its own, such as keeping an applied promo code when a bad code is entered, and Claude Code's tests now lock those choices in.
How I Ran the Test
The app is a small shop with a catalog, a cart, promo codes, tax and a checkout, defined in a spec of 12 requirements. Claude Code got the spec and one instruction in the pattern Anthropic's Claude Code best practices recommend: build it, write automated tests, run them, and keep fixing until they pass.
- Agent - Claude Code 2.1.327 running Claude Opus 5.5 at high effort, headless (claude -p), with file edits auto-accepted and npm, node and npx commands allowed.
- Isolation - an empty folder outside any repository, with skills, slash commands, MCP servers and Claude in Chrome switched off, so the only checks it had were the ones it wrote.
- Spec - requirements R1 to R12: out-of-stock items, the header cart count, quantity limits, an empty cart, two promo codes, tax rounded half-up to the cent, a $50.00 free-shipping threshold, a cart that survives a reload, checkout validation, order confirmation, stock updates and a rejected ZIP code.
- Independent checks - 12 Kane CLI objectives, one per requirement, written and hashed before the build started and never shown to Claude Code. Each ran against a freshly started server with a fresh Chrome profile, so no cart or stock carried over between runs.
- Seeded bugs - 8 one-line defects applied one at a time to the finished build, each followed by a full run of Claude Code's own suite.
Claude Code never saw the objectives, the seeded bugs or the phone-width check, so nothing it wrote could be shaped by them.
Claude Code's Build and Its 57 Tests
Claude Code finished in 45 turns and about 8 minutes, and reported 57 tests, all passing. I re-ran its suite on the delivered build and got the same count:
> mini-shop@1.0.0 test:unit
# tests 36
# pass 36
# fail 0
> mini-shop@1.0.0 test:e2e
Running 21 tests using 1 worker
21 passed (19.7s)- 36 unit and API tests - on Node's built-in test runner, covering pricing, rounding, promo codes, free shipping, form validation and the order API, including the 422 response for ZIP 00000 and the 409 for too little stock.
- 21 Playwright browser tests - in Chromium, walking R1 to R12 through the real pages against the real server. The prompt never asked for browser tests; Claude Code added them on its own.
- Integer cents - one shared pricing module serves both the browser and the server, so the cart total and the order total come from the same code.
- A test-only reset endpoint - the server can restore stock between browser tests, and the endpoint stays off unless an environment variable turns it on.
Its tests never failed against the app. The only red run in the build was a runner problem on Windows, where node --test failed to load the test folder as a module until Claude Code listed the three files instead. Every app test passed on its first run, so at this point the build offered no evidence that the suite could fail at all.
Did 12 Independent Checks Agree?
Yes. All 12 Kane CLI objectives passed against the delivered build, and each run ended on an assertion that matched its requirement.
| Requirement | What the Kane CLI objective checked | Result |
|---|---|---|
| R1 Out-of-stock item | "Out of stock" label, a disabled button, and Cart (0) after trying to click it | Pass |
| R2 Header count | Cart (3) after three adds | Pass |
| R3 Quantity limits | Desk Lamp x2 shows $69.98; x3 shows "only 2 in stock" and resets to 2 | Pass |
| R4 Empty cart | "Your cart is empty" and a disabled Checkout button | Pass |
| R5 Promo codes | FLAT5 error on a $10.00 cart, the invalid-code message, SAVE10 at -$1.00 | Pass |
| R6 Price summary | Subtotal $20.00, Discount -$2.00, Shipping $4.99, Tax $1.49, Total $24.48 | Pass |
| R7 Free shipping | Free at exactly $50.00, $4.99 at $40.00 | Pass |
| R8 Reload | Both items and the SAVE10 discount still there after a reload | Pass |
| R9 Checkout errors | Errors for empty fields, a bad email and a 3-digit ZIP | Pass |
| R10 Confirmation | "Order confirmed", an order number and a $25.02 total | Pass |
| R11 Stock update | Desk Lamp "Out of stock" after ordering both, and Cart (0) | Pass |
| R12 Rejected ZIP | A delivery error for ZIP 00000, still on checkout, Cart (1) | Pass |

- The rounding case - R6 was set up so that 8.25% tax on $18.00 comes to $1.485, which shows as $1.49 only if the code rounds half-up on exact cents. Claude Code's integer-cents pricing got it right.
- Run cost - the 12 objectives took 27 minutes and about 337 credits, one at a time against a local server, between 91 and 197 seconds each.
8 Seeded Bugs Against Claude Code's Suite
A suite that has only ever passed proves little, so I broke the finished build on purpose, one bug at a time, and re-ran all 57 of Claude Code's tests after each. This is mutation testing done by hand, with one-line regressions of the kind a later edit could introduce.
| Seeded bug | Requirement | Unit and API tests | Playwright tests |
|---|---|---|---|
| Tax truncated instead of rounded half-up | R6 | Caught (4 failed) | Caught (3 failed) |
| Free shipping uses > instead of >= at $50.00 | R7 | Caught (1 failed) | Missed |
| FLAT5 uses > instead of >= at $25.00 | R5 | Caught (1 failed) | Missed |
| Out-of-stock button left enabled | R1 | Missed | Caught (2 failed) |
| Header count stops refreshing on save | R2 | Missed | Caught (6 failed) |
| Checkout enabled on an empty cart | R4 | Missed | Caught (1 failed) |
| Server stops reducing stock | R11 | Caught (3 failed) | Caught (3 failed) |
| Cart cleared when the server rejects an order | R12 | Missed | Caught (2 failed) |
The suite failed on all 8, so a green run from it carried weight. Neither half managed that alone: the unit and API tests caught 4 of the 8, and the Playwright tests caught 6.
I also pointed the matching Kane CLI objective at 4 of the seeded builds, two the Playwright tests missed and two the unit tests missed:
- FLAT5 threshold (R5) - the objective passed, so it missed the bug. It only tries FLAT5 on a $10.00 cart, while Claude Code's unit test checks exactly $25.00.
- Free-shipping threshold (R7) - the objective failed with exit code 1, reporting that five Canvas Totes ($50.00) showed $4.99 shipping instead of Free. All 21 of Claude Code's Playwright tests had passed this bug.
- Header count (R2) - the objective passed, so it missed the bug, and the step log shows why. When the count did not change, the agent opened the cart, set Canvas Tote to 2 and went back to the catalog, where the page load refreshed the header to Cart (3). My objective asked for the final count, not an immediate update without leaving the page, so the gap is in how I wrote it.
- Cart cleared on rejection (R12) - the objective failed with exit code 1, reporting that after the ZIP delivery error the header showed Cart (0) while the item was still in the order summary. The unit tests had passed this bug.
- Discarded - a ninth bug, computing tax in floating-point dollars instead of integer cents, returned the correct tax on all 20,001 subtotals from $0.00 to $200.00, so it was not a bug and I did not count it.
Requirements Where Unit Tests Were Enough
For the pricing rules, Claude Code's unit tests did all the work. Both threshold bugs failed a unit test and passed every Playwright test, because the browser tests never built a cart at exactly the threshold.
- Promo and shipping thresholds (R5, R7) - only the unit tests checked the exact $25.00 and $50.00 boundaries, and only they failed when the boundary moved by a cent.
- Tax rounding (R6) - the unit tests pinned half-up rounding, including a 412.5-cent case that must round to 413.
- Stock after an order (R11) - the API tests caught the missing stock update without opening a browser.
- Where the browser was needed - the out-of-stock button, the header count, the empty-cart Checkout button and the cart kept after a rejected order (R1, R2, R4, R12) failed only in Playwright, because those bugs lived in page code the unit tests never load.
The split follows the usual line between unit testing and end-to-end testing: rules that live in one module, and what a page does with them.
Gaps Outside the Spec
Everything they missed sat outside the spec, which also means my 12 objectives missed it.
The Cart Breaks at Phone Width
Every one of Claude Code's 21 browser tests ran in a desktop Chrome window, and so did my Kane CLI objectives. Below about 520 pixels wide, the cart table overflows its card and the Remove buttons leave the screen, so a shopper on a Pixel 7 sees a grey sliver where each button should be.

- Widths checked - Remove was fully on screen at 520 pixels and wider, and off screen at 480, 430, 412, 390, 375 and 360 pixels.
- Its own suite caught it once asked - adding a single Pixel 7 project to the Playwright config Claude Code wrote made its own R4 test fail, 20 passed and 1 failed, because the line-total cell sat over the Remove button and took the click. Playwright's device emulation adds that project in one config entry.
- My objectives could not see it - Kane CLI's desktop window accepts widths from 800 pixels up, so a phone-width check needs a phone browser rather than a narrower desktop window.
To see the cart the way customers do, open it on real Android and iOS browsers in TestMu AI's real device cloud before release.
Decisions the Spec Left Open
Claude Code's final report listed where it filled gaps in the spec on its own, and its tests now assert each choice:
- A bad promo code leaves any code already applied in place, where the spec only said "apply nothing".
- Codes ignore case and surrounding spaces, so " save10 " works.
- FLAT5 drops off with its message if the cart later falls below $25.00, and the server applies the same rule.
- Add to cart on the catalog stops at the product's stock and shows "Only N in stock".
- The price summary is hidden on an empty cart.
None of these is wrong, but each is a product decision nobody signed off. The post Can Coding Agents Test Their Own Code calls this the failure self-testing cannot catch, because code and test share one reading of the spec; here the gaps were at least visible, because Claude Code reported them.
Note: Run spec-first checks on every build Claude Code hands you: Kane CLI by TestMu AI turns each requirement into a real-browser objective with a pass or fail and an evidence pack, so you can start free on your next build.
Changes to Make in Your Claude Code Workflow
Anthropic's best practices warn that "All tests pass" on its own tells you very little, and this run shows which parts of that sentence to check.
- Ask for unit and browser tests by name - each half missed bugs the other caught, so a prompt that only says "write tests" relies on the agent choosing both.
- Add a phone project to the Playwright config - one device entry would have failed its R4 test on the first run.
- Write acceptance checks from the spec, before the build - they cannot inherit the agent's reading of it. Mine still missed the FLAT5 boundary and let the agent reload past the header bug, so put exact threshold values and "without leaving the page" in them.
- Seed one bug before trusting a green suite - move a threshold or re-enable a disabled button, and confirm a test goes red.
- Read the agent's list of assumptions - its spec decisions are where misread requirements hide.
Kane CLI was the second opinion in this run. It runs natural-language objectives in a real Chrome browser, returns pass or fail with exit codes 0 to 3, and seals per-step screenshots, a HAR network log and console output into an evidence pack for each run.
Claude Code can call it from the terminal after its own suite, as the Claude Code Kane CLI verification guide shows, and the Claude Code app testing page covers the same loop in CI. For how it compares with Playwright MCP and Claude in Chrome, see the best E2E testing tool for Claude Code.
Conclusion
Pick the one requirement your app can least afford to break, write it as an objective with an exact expected value, and run it after Claude Code says the work is done. Claude Code's own suite caught every bug inside the spec here, so the second check earns its place on what the spec left out. The Kane CLI introduction covers sign-in and the first run.
npm install -g @testmuai/kane-cliTesting from the terminal? Kane CLI by TestMu AI lets SDETs and developers plan, write, run and debug tests from the command line.
Author
Prince Dewani is a Community Contributor at TestMu AI specializing in AI agents, software testing, QA, and SEO. He is certified in Selenium, Cypress, Playwright, Appium, Automation Testing, and KaneAI, and presented academic research on AI agents at PBCON-01. At TestMu AI, he has also carried out extensive cross-browser research on the support of modern web technologies such as WebGPU, WebAssembly, WebXR, WebGL2 and other web technologies, validating their compatibility and feature parity across major browsers and rendering engines through rigorous hands-on testing. Prince has hands-on experience building AI agent workflows using Anthropic Claude, Google Antigravity, n8n, LangChain, and other agentic frameworks, and works regularly with MCP and A2A protocols. He shares his work with 5,500+ QA engineers, developers, DevOps experts, tech leaders, and AI agent practitioners on LinkedIn.
Reviewer
Shravan Mahajan is a Software Engineer at TestMu AI building Kane CLI, the command-line tool that runs browser automation from the terminal, describing flows in natural language that execute in a real Chrome browser and return pass or fail with shareable proof. He has an experience of 6 years in the Technical industry. His top skills are JavaScript, React.js, and full-stack development. At Fractal he built automated data pipelines with T-SQL, SSIS, Python, and Azure. He is also a Microsoft Certified Azure Data Engineer Associate.
Claude Code Tests FAQs
Did you find this page helpful?
More Related Blogs
TestMu AI forEnterprise
Get access to solutions built on Enterprise
grade security, privacy, & compliance
- Advanced access controls
- Advanced data retention rules
- Advanced Local Testing
- Premium Support options
- Early access to beta features
- Private Slack Channel
- Unlimited Manual Accessibility DevTools Tests


















