How a Retail Bank Replaced Self-Reported Coverage With Machine-Read Evidence

Overview of the Retail Bank Coverage Review

A retail bank reported 87% test coverage every quarter. The figure was typed into a slide, not read from any system.

The bank ran Kane CLI from TestMu AI against its existing suite of 4,200 tests and scored each one on what it verified. 41% of the suite proved little or nothing. Coverage is now a reading produced by the system, backed by evidence that internal audit can inspect.

A Retail Bank That Answers to Audit and Regulators

The bank is one of the country's large, full-service retail banking groups. It serves millions of customers across deposits, cards, lending and wealth products through its branch network and digital channels.

Its digital banking division, a 240-person engineering organization supported by 38 SDETs, runs the mobile and web journeys customers use to check balances, move money and manage accounts. Internal audit, external regulators and customers all periodically require demonstrable proof, not assurance, that the systems behind those journeys are properly tested.

The Challenge: An 87% Coverage Figure Nobody Could Inspect

The division ran 4,200 automated UI and API tests every night against its retail banking platform. Dashboards were green, and the quarterly business review reported 87% coverage.

The team set its coverage target internally and maintained the mapping of tests to requirements by hand. A review commissioned by a new Head of Quality Engineering then examined the suite itself and found three patterns that ran without verifying anything:

The coverage report measured execution, not proof. A test that ran to completion and a test that verified an acceptance criterion looked identical on every dashboard the bank had, so each result was a claim internal audit had to accept on trust.

How an Internal Audit Question Led the Bank to Kane CLI

Internal audit asked how the division knew its tests proved the requirements they were mapped to. The honest answer was that the mapping asserted it rather than demonstrated it.

The team evaluated Kane CLI because it derives coverage from what the system executed and verified, instead of from a value someone enters. Each result carries a score, each score is backed by an artifact, and each artifact is linked to the requirement it verifies.

How Kane CLI Scored the Existing Suite Without a Rewrite

The team applied Kane CLI to the suite it already had rather than rewriting tests at the outset. The first full scan scored every test on what it verified and separated tests that confirmed a real acceptance criterion from tests that only ran to completion. Null checks, unverified soft asserts and scroll-and-count steps were all marked as execution without verification.

The division then rebuilt its coverage picture on the Kane CLI chain of evidence:

The coverage figure became a reading produced by the system rather than a value maintained by a person. Any stakeholder can move from a passing result to the score behind it, the artifact behind the score, and the requirement behind the artifact.

Results: 41% of the Test Suite Proved Little or Nothing

The first machine-read coverage figure was substantially lower than the 87% the division had been reporting. 41% of the suite was found to prove little or nothing.

It was also the first coverage figure the division did not have to defend, because anyone could inspect it directly. Over the following quarter, the team retired or repaired the weakest tests and rebuilt coverage against the requirements that mattered, with supporting evidence attached.

A test that runs and a test that proves something used to look the same to us. They no longer do.

Senior SDET

Digital banking platform team, Retail bank

Each Passing Bank Test Now Traces Back to a Requirement

Every passing result now has a score, an artifact and a requirement behind it. When audit asks how the division knows its controls are tested, the team opens the evidence instead of a spreadsheet.

Ready to improve your testing? Schedule a demo to see how TestMu AI can help your team achieve similar results.