Ranking Test Suites by What They Caught [Testμ 2026]

The Metric Nobody Trusts

Those are separate questions, and the session is about answering the second one. He has spent over two decades in quality engineering and now leads test teams for streaming platforms serving millions of viewers, across test automation, performance engineering and DevOps integration.

The promise he makes at the start is that every test in your suite ends up with one of three verdicts: proven, duplicate or unproven.

He also asks the audience to hold their first reaction to the numbers loosely, because the interpretation gets more complicated than it looks. That warning turns out to be the most useful part of the talk.

Two Tests, Same Coverage

The problem in miniature is two tests sitting side by side in the same report. Test A has caught three real incidents. Test B has never failed on anything.

Both report exactly the same test coverage. The coverage dashboard cannot tell them apart, because it was never built to.

That is the whole case for a second metric sitting alongside the first, rather than a complaint about coverage as a measurement.

Proven, Duplicate, Unproven

The method fits in one sentence: replay every logged incident against every candidate test, and give each test a verdict. What a test scores has nothing to do with its coverage percentage or its assertion count.

• Proven - unique historical catch evidence. The test failed on at least one real incident that no earlier test had already caught.
• Duplicate - a real signal, but always after an upstream test had already flagged the same thing. Not automatically waste.
• Unproven - healthy coverage with zero historical catches, flagged for owner review rather than deleted on sight.

He is insistent that duplicate and unproven are different problems even though they look identical on a dashboard. A duplicate is redundant by design, because some tests fire earlier in a different stage on purpose.

An unproven test means only that it has no track record yet. Neither group gets deleted automatically, and both go to the test owner for a conversation.

Record, Revert, Replay, Tally

The mechanism runs in four steps, with the pre-incident state pulled from timestamped deployment history.

• Record - the incident gets written to a database with its component and timestamp.
• Revert - the system is rolled back to its state just before that incident’s first symptom.
• Replay - every candidate test runs against that reverted state.
• Tally - the failures are counted, and that tally becomes the verdict.

The detail that carries the method is cross-replay. Every test is replayed against every incident, not only the one it happens to have been written for.

Without that, the exercise is a retrospective pat on the back. With it, a verdict reflects how often a test would have failed on something that actually happened.

The 4,200 Test Replay

The case study covers roughly 4,200 automated test cases replayed against every logged incident in an 18 month window.

• 80% proven - failed on at least one incident that no earlier test had already caught.
• 8% duplicate - only ever caught what had already been flagged upstream.
• 12% unproven - never failed on a single logged incident across the whole 18 months.

One test in eight, then, has no catch history at all, and about a fifth of the suite carries no unique catch evidence once duplicates are added in.

He spends the rest of the session arguing against the obvious conclusion.

The Preventive Blind Spot

The first limitation is structural rather than an edge case. A reactive test catches an incident after it happens, which is exactly what this method detects, so it scores proven correctly.

A preventive test blocks the bug before it can become a loggable incident. By construction it can never accumulate catch history, so it scores unproven incorrectly.

Both tests can be equally valuable, and only one of them will ever look valuable to the score.

His example is a regression test for a playback stall fix that appears nowhere in 12 to 18 months of incident logs. It is absent because it keeps failing in CI on every attempted regression, stopping the bug before it reaches production.

Backwards scoring calls that test unproven. An owner review caught the mismatch in minutes, and an automated cutoff based on the score alone would have deleted it.

The Incident Log Limit

The second limitation is about your data rather than your tests. An incident that was never logged cannot be caught by this method, no matter how good the test guarding against it is.

A team with a thin incident history will therefore see every test in its suite look unproven. That result describes their logging discipline, not the quality of their tests.

Which is why he reads every verdict the method produces as a floor on proven value and never as a ceiling on total value.

Flags, Not Delete Scripts

A verdict gets computed for every test, and the unproven ones get flagged. The flag goes to the test owner.

The owner then reaches one of two outcomes. Confirmed dead weight has no catch history and no plausible preventive story, and gets retired. A confirmed preventive test is doing its job upstream of where the incident log even starts, and gets relabelled.

That is the difference between a score used as evidence and a score used as an execution order. In the case study, none of the flagged tests were removed automatically.

Against Coverage Thresholds

His objection to coverage thresholds is about what they incentivise. They push you to write more tests, and not to write tests that catch anything.

He put the two side by side across three attributes.

Coverage counts the test cases you wrote against the requirements, and maps cleanly to those requirements when they are mapped well. Incident prevention is a separate measurement that runs through defect density and test effectiveness.

His position is that both questions deserve answering, separately.

Neighbouring Methods

He spent time distinguishing backwards scoring from techniques it gets confused with, and framed all of them as complementary rather than competing.

• Mutation testing - introduces synthetic bugs proactively and exhaustively, where backwards scoring works from real incidents in retrospect. A mature suite probably wants both, one synthetic and one empirical.
• Test impact analysis - predicts which tests to run for a given change when time is limited. Backwards scoring asks which tests are worth having at all.
• Chaos engineering - injects failures into a live system to test resilience under pressure, rather than checking whether the existing suite would have caught failures that already happened.
• Observability and monitoring - Datadog, Splunk or New Relic tell you about an incident as it happens and trace it back to the source, producing a ticket. That is retrospective on the critical path, and it is where the incident records this method replays come from.

The distinction he most wanted to land is between a coverage gap and an unproven test. A coverage gap means no test exists for what broke, while an unproven test exists and runs and has caught nothing.

An incident with no test at all is a coverage gap and a different finding, calling for a test to be written rather than an existing one to be reviewed.

Mapping and Flaky Tests

Mapping an incident to a test deterministically starts from the component and the timestamp. From the timestamp you pull the coverage and replay against the prefix commits, then find the gaps in the deployment history.

Those gaps get excluded rather than approximated, which is what keeps the mapping deterministic instead of inferred.

Flaky tests are handled by exclusion. Only reproducible counts are scored as catches, and a single flaky flicker is excluded from the model entirely.

His instruction on those is to go to the root cause and establish whether the failure was a code issue or something trivial such as a changed locator at the UI level.

On bias, he conceded the point rather than defending it. Hindsight bias exists in a method built on a track record, the real limitation being the preventive blind spot, and neither is deliberate.

Q & A Session

The Q&A box was busy through the talk, and three questions made it into the time remaining.

• For the tests that did not flag anything, were they discarded or adapted for possible viability?

  Partha Sarathi Samal: They were not discarded. They were adapted and adjusted instead, and no test is set aside on the strength of its flag. He returned to logging discipline as a prerequisite for the whole method, since an unlogged incident is not hidden value in a test, it is a missing record.

• Could an agent that makes humans more effective count as successful even if it completes fewer tasks autonomously?

  Partha Sarathi Samal: Yes, with a human in the loop, and the deciding factor is grounding. A large language model generates a great deal of output, so what matters is whether it is grounded in the knowledge you have put into the system, at which point it succeeds on the product outcome and not only in your view. On task count against accuracy, he would spin up parallel agents to recover throughput, and keep a human for oversight because the subject matter expertise sits there.

• You advise flagging unproven tests for owner review instead of auto-deleting at scale, so how do you stop that review becoming a bottleneck for test owners?

  Partha Sarathi Samal: The review queue is smaller than it sounds, because the suite is not uniformly unproven and the proven set needs no attention. He expects friction early on, then a queue that shrinks as teams learn from the mistakes the first pass surfaces and stop repeating them, spread across several people rather than one owner. He also expects AI to absorb part of the load once it is grounded in a baseline of expected against actual results, which is the same framing he applied to the previous question.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.