10 Root Causes Behind a Failing Test Suite [Testμ 2026]

The Repository Nobody Trusts

His flaky tests were the familiar kind. His fragile ones had a specific trigger: a library update, not even a package update, changed a chain item near the root of the DOM and almost every test started failing at once.

Tests that break when the page structure shifts are tests bound to the structure rather than the behaviour. A small change was enough to take the suite down.

The forgotten ones are the saddest category. Some had been failing for so long that everyone had lost faith in them, and nobody wanted to fix them any more, because fixing one bought a few days before it failed again.

The Real Cost of Flakiness

The immediate cost is the pipeline stopping for a failure that is not a product bug. Somebody has to run the root cause analysis to establish whether it is real or a false alarm caused by an unhandled wait, then maintain the test afterwards.

That time comes out of new development, and it compounds. With enough failing tests there is never room to fix them all, so the backlog and the failure count grow together.

The cost he rates highest is not measured in hours. People stop believing the results, and reliability is precisely the property of a suite whose pass means pass.

The mirror image of a false alarm is worse. When there is a real bug and the test does not find it, that is a silent horror case, and it surfaces later as an escaped bug in production in front of customers.

Tests at the Wrong Level

Asked why the pipeline took four hours, he did not have to look far. There were too many test cases, which he admits is an unusual complaint from a test lead.

His objection is to volume without value. Reviewing the system-level suite, he found most of it had no business being at the system level and belonged at component or unit level instead.

His example is the end-to-end test that verifies an element is on the page. If a user needs that element, the user needs it inside a flow, so the test should click the button and observe what happens rather than confirm it exists.

Volume also costs more than time. At 500 tests the risk of failure rises with the count, and some of them start conflicting with each other, which is a problem he returns to later.

Shortcuts and Repeated Setup

The second cause was work the suite repeated because nobody had questioned it. Every single test logged into the system, when logging in once and reusing the session information would do.

His second example is sharper. Their application has a dropdown that switches between live and draft mode, and the same switch happens by appending a segment to the URL.

The dropdown itself deserves a test. It does not deserve to be exercised in every other test that merely needs draft mode, particularly when a dropdown is exactly the kind of control that turns flaky.

The third cause is the same waste inside a single spec file. Several tests each performed the same login, navigation and setup before making one assertion, where one test could do the setup once and make all the assertions.

He was careful not to turn that into a rule. Merging everything into one enormous test is its own problem, and finding the point where nothing repeats but the test stays maintainable is judgment rather than policy.

The Infrastructure Question

The fourth cause was the stack itself. Their setup ran Selenium tests orchestrated by WebdriverIO, which called out to BrowserStack to raise browsers.

His criticism is about fit rather than quality, and he said so explicitly. The device cloud in that chain earns its place when you are testing native mobile applications, and his team was not testing mobile apps at all.

Removing a layer from the communication chain cut execution time on its own. They later moved the framework to Playwright, which drives the browsers directly.

A second decision came with it. They stopped running every test against every browser on the grounds that modern browsers support most features, keeping the full suite on one browser and reserving cross-browser runs for the smoke or high-priority set on a nightly or weekly cadence.

Readability Beats Reusability

Causes five through seven all produce the same symptom, which is a test nobody can read. The first is over-abstraction: a test whose body is one line calling a function, which calls other functions.

Debugging that is a chain of jumps. The report says line 15, line 15 is a function call, that leads to line 25, and line 25 leads somewhere else.

His ranking is the part worth arguing with your team about. In test code, readability, understandability and maintainability win over reusability, and repeating the same assertion across tests is acceptable because the assertion is the intent of the test.

The sixth cause is hidden logic, where verification steps live in helper or page files two hops away from the spec. Test intention should be clear from reading the test, starting with its name, and he has seen names that describe something other than what the test does.

The seventh is the god function, which takes many parameters and branches into different states to cover every scenario. At the call site nobody can tell which branch a set of parameters triggers, and when it fails there is no small unit to fix.

Stability Before Merge

The eighth cause is the one that keeps producing the others. Tests were merged that had never been shown to be stable, and started failing the same night.

Asked about it, engineers said the test passed when they ran it. It always does, and the pipeline differs in timing, execution speed and how long a page or its components take to load.

The other difference is company. A test alone behaves differently from the same test inside a suite, and his example is two tests that cannot coexist: one resets every filter to see the full product list while another sets filters to see a subset.

Run in parallel they fight each other. The fix is either an execution pattern that orders them or a merge into one test that filters and then resets.

His practical advice is to run the branch on the pipeline several times before merging, because some failures only appear under conditions that do not reproduce on the first attempt.

He also named the pressure behind it. Engineers told him the team expected the merge today, which left him asking whether they wanted him to skip the code review or the quality checks, and his answer is to start QA work when development starts rather than compressing it into the last two days of the sprint.

Silence and Invisibility

The ninth cause is the source of those silent horror cases. Liberal use of try and catch means a query fails, the error is swallowed, and the test carries on to the next step without complaining.

He does not ban the construct. He asks for a valid reason for each one, and for attention to whether the catch is genuinely handling a failure case or just hiding it.

The tenth covers duplication and reporting together. Fixes had to be applied in several places because the same interaction code was scattered across spec files, which is what a properly applied page object model prevents by keeping methods in page or helper files and locators in component classes.

Reporting was the other half. He showed a screenshot of their pipeline and asked whether the audience could see that a test had failed, which nobody could, and neither could his team.

People were searching pipeline logs for the word failure. They now publish a report stating what passed, what failed, what was flaky and what was retried, built on the reporters that ship with the framework or on open source alternatives.

His last point on reporting is about retention. Reports that vanish with the ephemeral environment take the history with them, so screenshots, traces and logs should be stored long enough to compare against the next failure.

Ten Fixes, Summarised

He closed the diagnosis by compressing it into a checklist his team now reviews against.

• Define and automate at the correct level - respect the test pyramid, and keep component checks out of system-level tests.
• Use shortcuts - reuse a session rather than logging in for every test, and reach a state by URL where the UI path is not what is under test.
• Design test cases efficiently - stop repeating the same setup steps across a spec file.
• Choose tools and infrastructure that match the requirement - do not raise browsers or devices the suite does not need.
• Avoid over-abstraction - a test should be readable without following a chain of function calls.
• Keep test logic and naming explicit - the intention should be clear from the test itself, starting with its name.
• Avoid god functions - one function, one responsibility, no conditional verification branches.
• Prove stability before merging - run the branch on the pipeline several times, not once on a laptop.
• Be deliberate about try and catch - never swallow an error silently.
• Eliminate duplication and improve reporting - apply the pattern consistently, and make results visible and traceable.

Three AI Skills

Out of those lessons he built three skills, one for each stage of the lifecycle where the mistakes were being made.

• Test definition - connects through MCP servers to the issue tracker, reads the feature details, acceptance criteria and requirements, then applies decision criteria for what belongs at component, integration or system level and writes the cases into a markdown template.
• Implementation - reads that markdown alongside a coding guideline covering the page object model, expected file layout and the anti-patterns to avoid, and generates the spec, page and component files.
• Pull request readiness - checks the result against the rule set: no parameterised or conditional verification methods, no duplication, nothing reimplemented that already exists in the repository, single responsibility per function, and no pass-through abstraction layers.

The first skill fixed a process problem as much as a technical one. Test reviews used to happen in text files, spreadsheets, chat messages and email, none of it trackable, and putting the cases in the repository as a pull request means the discussion and its history survive the merge.

The third plays to what machines are good at. In a large repository nobody can quickly tell whether the helper being added already exists somewhere, and the review rounds his team used to run have measurably shortened.

The Risks of Leaning on AI

He ended the demonstration with four cautions rather than a recommendation.

• Hallucination - use it, but review the output rather than trusting it blindly.
• Over-reliance - generate everything for long enough and you stop knowing what your own repository does, which he considers the more dangerous failure.
• Performance - ask whether there is a better way and the answer is often yes, which means the first version was not the efficient one.
• Cost - every skill run and every query spends money, and keeping that down is the team’s responsibility rather than the employer’s.

Q & A Session

Three audience questions closed the session.

• What is the first warning sign an automation project is dying, and how much of flakiness is a tooling problem versus a people problem?

  Mesut Durukal: The first sign is people losing trust. When he joined the project the complaints arrived immediately, that tests took too long and failed too often, which told him the results were no longer being relied on. The measurable version is the number of failing tests, and it only means something if the history is kept, because a single latest run says nothing about the pass and failure percentages over time. On tooling versus people, he takes people to mean human engineering and comes down firmly on that side. Moving from Selenium to Playwright or Cypress does not make flakiness disappear, because it is mostly a product of how the tests are implemented.

• What strategies eliminate flakiness in a CI/CD pipeline, apart from running across multiple workers in parallel?

  Mesut Durukal: Parallel workers do not eliminate it. What helps is running a test several times before merging to confirm it is stable rather than lucky, then following the implementation practices and avoiding the anti-patterns covered in the session.

• Under intense delivery pressure, what secures organizational buy-in to pause and address technical debt in automation?

  Mesut Durukal: He was candid that he is living this problem rather than having solved it. Pausing feature delivery to pay down debt has not worked for him, and he does not expect it to, because acceleration is what product leaders are measured on and they will not stop. What works instead is reducing the debt alongside the work, whenever bandwidth appears, which is only possible if automation tasks are shared. If the quality assurance members are the only people automating, the arithmetic never adds up, and he has found that QA engineers themselves sometimes resist sharing because they feel ownership of those tasks. His position is that quality is a shared responsibility and developers jumping in is what creates the bandwidth to deal with the debt at all.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.