Why a Payments Platform Stopped Trusting Retried Test Passes

Overview of the Payments Platform Retry Audit

A payments platform let its test retries turn real failures green, and a checkout race condition reached production.

After the incident, the team adopted Kane CLI from TestMu AI so that no green build goes unexamined. The first audit of retried passes found more than 60 tests that passed only on a second attempt. Nine of them were masking genuine race conditions, and all nine were fixed before they reached customers.

A Series C Payments Platform in the Settlement Path

The company processes card and bank transfer payments for online merchants and sits in the settlement path between shoppers and the businesses they buy from. Its 150-person engineering organization, within a company of roughly 400, owns everything from checkout integrations to the reconciliation logic behind them.

That scope raises the cost of a defect. A race condition in this codebase can duplicate a charge, and a bug that reaches production means money moving incorrectly for a merchant and their customers. That exposure turned a routine effort to reduce flaky tests into a review of what a passing test actually establishes.

The Challenge: Retries Hid a Checkout Race Condition

Eighteen months earlier, the team had set retries: 2 in its Playwright configuration. Flaky tests were producing enough red builds to hide genuine failures, and retries removed the noise.

A test could now fail on the first attempt, pass on the second, and the pipeline would report a green build. A genuine race condition in the checkout flow followed that path. It failed once, passed on retry, shipped, and surfaced during a merchant's peak sales period as intermittently duplicated payment attempts. The postmortem found the test that had detected it, and found it had been detecting it for weeks.

The same postmortem exposed two blind spots in the suite:

How a Production Incident Led the Team to Kane CLI

After the incident, the engineering director's requirement was that no green build should go unexamined.

The team adopted Kane CLI because its execution evidence makes each run inspectable: what every step did to the application, why a failure occurred, and whether a pass was earned or achieved on a second attempt. A pass on retry becomes a signal in its own right, with the evidence from the first failing attempt kept rather than overwritten.

How Kane CLI Classifies Failures and Flags Retried Passes

Evidence-first execution changed what a red and a green result each mean in practice.

Engineers no longer open several tools to work out which category a failure belongs to, and an action that changes nothing no longer passes through to the end of the run.

Results: 9 Race Conditions Fixed Before Reaching Customers

The first full audit of retried passes identified more than 60 tests that passed only on a second attempt. Nine were masking genuine product race conditions of the same type as the one that had reached production. All nine were resolved before they affected customers.

Triage time fell by roughly two-thirds, because each failure now arrives with its cause attached. The team has retired re-running the suite as a diagnostic step.

A red result used to mean the cause could be anything. It now tells me something specific before I open the report.

SDET

Payments platform

Payment Builds Now Separate First-Try and Retried Passes

A pass on the first attempt and a pass on retry no longer look the same in the build report. Each red result names its cause, and each step proves it changed the application.

Ready to improve your testing? Schedule a demo to see how TestMu AI can help your team achieve similar results.