Why Coding Agents Write Flaky Tests, and How to Make Them Deterministic

Why Do AI-Generated Tests Flake?

AI-generated tests flake for three separate reasons, and each one needs a different fix. We refer to them as the three sources of agent-induced flakiness: copied flake patterns, app drift, and model in the verdict. To say the AI is flaky is really how teams end up just re-running their pipelines rather than actually fixing anything.

Source 1 is the one research measures. A fourth behaviour makes all three worse: agents react to red builds by retrying, skipping, quarantining or "healing" tests, which turns a signal into silence.

A separate failure to mention is the case of tests that pass every time on a broken app. That's a false green, not flakiness. The very fact that AI-generated tests will pass when your app is broken is the textbook example of that.

What Does the Data Actually Say About AI-Written Tests and Flakiness?

Every study we've seen points in that direction. The generated tests flake just as much as the handwritten ones. That's what the developers themselves are saying.

What developers are saying (TestMu AI analysis of Hacker News). We've done an analysis of 16,530 Hacker News comments from 1st Jan 2023 to 4th Nov 2026 (pulled from the HN Algolia API via 38 queries). That gave us 779 candidates and 350 confirmed testing complaints after AI-assisted review. Of those 350:

• Rerun culture (re-running until green, quarantining, losing trust in red builds) is the number 1 pain point with 64 complaints (18.3%).
• Brittle tests: 51 (14.6%); Timing and concurrency problems: 35 (10.0%)
• AI or agent written/AI powered tests being flaky: 15 (4.3%) - 3 in 2023, 1 in 2024, 4 in 2025, 7 in 2026.

We read all 15 by hand, not just the labels. They split three ways.

Two describe AI-written or model-judged tests flaking in practice:

• A user was able to help a mate whose AI-generated tests were flaking. It was down to a boolean that had been flipped, behind a comment which was confidently wrong. Fixing just the flake would have left the test failing every time. (HN, May 2026)
• A developer's tests (where the LLM picks a tool from a list of tools in the text) are flaky, as the model will occasionally get the argument types wrong (HN, April 2026). That's source 3 in its purest form.

Four describe neighbouring failures:

• A QA agent built on Claude Sonnet 3.5 and fed screenshots was very confident in reporting a component which was obviously broken as being working. That's a false green, not flakiness (HN, Sep 2025).
• A tool author made the observation that coding agents are very much prone to breaking Playwright tests (e.g. by renaming a button), so a test which was previously green will suddenly become red. That's brittleness, source 2 (HN, Jul 2026).
• An engineering lead spotted the flaky integration tests as one of the problems in a number of PRs from engineers who had started to use the AI tool. They didn't attribute the flakiness to the AI (HN, Feb 2025).
• A lone dev was concerned by Claude choosing which of the flaky tests to isolate. The same agent then went on to find the race condition behind them (HN, Jun 2026).

The other nine are either opinions/predictions that AI-driven testing would be unreliable or slow, or they're just discussions of the issue of nondeterminism in agent evals and sandboxes with no actual AI-written test flaking.

The HN data confirms it's a very real and current issue. The research below has the figures.

What researchers measured:

Two things follow. First, the dominant LLM failure in the database study was order dependence, which is cheap to catch and cheap to fix (see the Python example below).

Second, flakiness spreads through context: if your suite already flakes, the agent learns the pattern from it. Cleaning the existing suite is a prerequisite for trusting agent-written additions.

What Is a Deterministic Test?

A deterministic test will always give the same result when run against the same code, same data and same environment. The outcome is purely a function of the behaviour under test. Timing, ordering, randomness, the clock and all the other 'leftovers' from other tests have no bearing on it whatsoever.

Martin Fowler's 2011 definition of the antithesis remains valid: a non-deterministic test will pass some of the time and fail some of the time with no change to the code, the tests or the environment (Eradicating Non-Determinism in Tests).

Flakiness was already a cost prior to agents. Google reported that around 1.5% of their test runs would be flaky, about 16% of their tests were flaky, and roughly 84% of the pass-to-fail transitions they observed were due to a flaky test (Google Testing Blog, May 2016).

With coding agents in the loop, a deterministic test needs two extra properties:

• The verdict is determined by the criteria that the human has written, not by the model's reading of the screen on that run.
• Every run leaves behind the kind of evidence that a person could go back and open at a later stage (steps, screenshots, logs, verdict) so that "it passed" can be ticked rather than just accepted on trust.

A brittle test can be perfectly deterministic. It will fail every time after a completely innocuous UI change. That's a different bug with a different fix, and that's precisely the scenario our flaky test guide is designed to deal with.

How Do You Go About Checking That an Agent's Test Is Deterministic?

Run it loads of times before you merge it and treat a single inconsistent result as a failure. One green run of a new agent-written test tells you absolutely nothing.

Playwright: repeat each new or altered spec 20 times in parallel. A failure in any one of the repeats will cause the command to fail.

npx playwright test tests/checkout.spec.ts --repeat-each=20 --workers=4

In CI, allow retries but fail the build if a test only passes on retry (so that flakes are visible rather than just being swallowed).

npx playwright test --retries=2 --fail-on-flaky-tests

Both flags are in the Playwright CLI reference. --fail-on-flaky-tests will cause the run to fail if any test is flagged as flaky, and it's off by default.

pytest: install pytest-repeat and run the new test file 20 times.

pip install pytest-repeat
pytest --count=20 tests/test_orders.py

Make it a rule for the agent. Add this to CLAUDE.md or AGENTS.md: "Any test you add or edit must pass --repeat-each=20 before you report the task as complete. Never add retries, skips or longer timeouts to get a test to pass."

Instruction files are advisory, so back this up with a CI check. This guide covers the hooks for Claude Code, Cursor, Copilot and Codex.

Take the verdict away from the agent that wrote the code.

Note: Kane CLI by TestMu AI runs the user flow in a real browser. There's a separate evaluation step that determines pass/fail, and every run generates an evidence pack. Your first run is simply: install, log in, run one objective, and you get an exit code that CI can gate on.

How Do You Fix the Flake Patterns the Agents Produce?

The vast majority of agent-written flake comes down to four patterns: fixed waits, structural selectors, order assumptions, and uncontrolled time. Each of those has a one-line fix.

Fixed waits and structural selectors (Playwright). This is the kind of test that agents produce when they're writing a test from the DOM that they've just generated:

// Flaky: guesses how long the page takes, and couples to a CSS class
await page.goto('/checkout');
await page.click('.btn-primary');
await page.waitForTimeout(3000);
expect(await page.locator('.total').textContent()).toBe('$42.00');

The deterministic version waits on state, looks at what the user sees, and then freezes the clock so that date-dependent pricing or banners, etc. won't have moved.

import { test, expect } from '@playwright/test';

test('order confirmation shows the total', async ({ page }) => {
  await page.clock.setFixedTime(new Date('2026-11-04T10:00:00Z'));
  await page.goto('/checkout');
  await page.getByRole('button', { name: 'Place order' }).click();
  await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible();
  await expect(page.getByTestId('order-total')).toHaveText('$42.00');
});

expect(...).toHaveText() will keep retrying until the condition is met or it times out, as opposed to a one-off textContent() read. page.clock.setFixedTime is the recommended way to pin Date.now() (Playwright clock docs). More on suite hygiene: Playwright best practices and Playwright flaky tests.

Order assumptions (Python). This is the pattern behind 63% of LLM-generated flaky tests in the database study:

# Flaky: assumes row order the database does not guarantee
assert [o.id for o in repo.list_orders()] == [1, 2, 3]

# Deterministic: compare order-independent values
assert sorted(o.id for o in repo.list_orders()) == [1, 2, 3]

If order is part of the contract, then include the ORDER BY in the query and keep the ordered assertion. Don't just paper over that in the test.

Randomness and shared state. Seed every random generator the test touches, give each test its own data, and never let an agent-written test read files or rows that another test has written. Gruber et al. found that generated flaky tests are more often due to randomness than developer-written ones, and the agent test preprint flagged file I/O.

Should an Agent Decide Which Flaky Tests to Retry, Quarantine or Heal?

No. The agent can propose a retry, a quarantine, or a heal, but the human has to approve each one as a diff. Each of those actions changes what the suite proves, and the agent has a direct incentive to turn red into green.

Flakiness is more expensive with agents than it is with humans. A human learns which reds to just ignore. An agent will act on every red, so a flaky suite will be chasing phantom fixes, and a quarantine habit will mean it will just hide the real ones.

The HN developer above was quite rightly concerned that their agent was the one choosing which flaky tests to isolate, even though it was then the one that found the actual race condition.

The tooling already has that capability. Playwright's healer agent will re-run the failing step and propose a patch. Its documented output is that it will simply skip a test if the healer thinks the functionality is broken (Playwright Test Agents).

That's a sensible approach, provided that someone looks at the patch. An unreviewed skip is effectively a loss of coverage.

This is the standard we have for every agent-written change (the agent that wrote the code doesn't grade it). Can coding agents test their own code? That's the full argument for independent verification.

Agent test merge checklist:

• Repeat-run 20x. Every test that the agent added or edited passes --repeat-each=20 (or pytest --count=20).
• No new retries, skips, or timeouts. Any added retries, test.skip, quarantine tag, or longer timeout will fail review unless a human approves it.
• Human-owned criteria. The acceptance criteria come from the ticket or a human-written test, and the agent didn't touch them.
• Evidence is all there. The run that demonstrated the change left steps, screenshots, logs, and a verdict that a reviewer can open.
• Heals are reviewed as diffs. Any self-heal or locator patch will be shown as a diff. A heal that changes what a test asserts is rejected.

How Do You Keep the Verdict Away From the Coding Agent?

Split the jobs. The coding agent writes the code and proposes the tests. An independent verifier then replays a fixed path and checks that the acceptance criteria are met against a human-approved version.

TestMu AI built Kane CLI to be that verifier, and its llms.txt describes it as a validation layer for coding agents. Our own Verification Bench (October 2026, designed, run and graded by TestMu AI) measured the gap: Claude Code driving Playwright MCP reported a pass which the page contradicted in 16.8% of runs; Kane CLI did so in 0.8%. Three steps:

1. Install it and give your agent the skill.

npm install -g @testmuai/kane-cli
kane-cli login
npx @testmuai/kane-cli-skill   # Claude Code, Codex CLI, Gemini CLI

Cursor and GitHub Copilot both read the Kane CLI AGENTS.md.

2. Following a change, the agent runs one objective and acts on the exit code.

kane-cli run "Go to http://localhost:3000/checkout, place an order with the test card, then verify the confirmation page shows the order total" --agent --headless --timeout 180

Exit 0 = passed, 1 = failed, 2 = didn't start and 3 = timeout. A pass means all the acceptance criteria are met, and there is a separate assessment step (the Kane CLI) which makes that call.

This is an ad hoc run for the build loop (it plans its path with a model each time). Gating is step 3's responsibility.

3. Save it, then gate on the saved test with healing off.

kane-cli run "<same objective>" --agent --headless --name checkout
kane-cli testmd run .testmuai/tests/checkout_test.md --agent --headless --no-adaptive-heal

--name writes .testmuai/tests/checkout_test.md and an output-checkout/ recordings folder alongside it. Move both of those into your repo and commit them. A saved Kane CLI test will simply replay its exact steps from the recording with zero LLM cost.

Healing is a toggle that you can turn on and off. It's on by default. When it's on (the default), if a step no longer matches, it will be re-authored from that step (up to 3 times) and then the whole test.

The acceptance criteria are never changed in any case. When --no-adaptive-heal is used, that step will fail the run rather than being re-planned, so the path that the test takes on a given commit is fixed.

Full walkthrough including CI gating and evidence packs: Kane CLI for coding agents. For E2E specific failure modes, see 12 ways AI agents fail at end-to-end testing. Tool-specific pages: Claude Code app testing and Cursor app testing.

Which Verification Approach Gives the Same Result Every Run?

Only the approaches that simply replay a fixed path and remove the verdict from the agent that wrote the code give a stable answer on the same commit. A saved Kane CLI test run with --no-adaptive-heal does exactly that: it replays the recorded path, fails on any step that no longer matches, and then has a separate assessment step to make the call. The facts around the other tools were checked in their docs on 4th Nov 2026.

Who Needs an Independent Verifier and Who Doesn't?

You need one when coding agents are shipping changes to user-facing flows (checkout, sign up, billing, etc.) as the changes to those flows need to be validated by someone who didn't write the change.

No, you don't need one for library code, parsers and pure functions. Fast unit tests, a seeded property-based suite and a repeat run check will do.

A team that's already built up a hand-written Playwright suite with a measured, near-zero flake rate and has a process of reviewing every agent edit to it has the discipline. Playwright is the right tool when you want to own the code and need the primitives (e.g., network interception), and many teams run both.

What no team can afford to skip is the standard: run every agent-written test through a repeat run and never let the agent edit its own pass criteria.

Start with the agent-written test that your team re-runs most frequently. Run it with --repeat-each=20. If it fails at all, then fix it using the patterns above.

If the flow is important to users, then write the acceptance criterion as a single sentence and have a separate verifier make the call. Put the Kane CLI on your next agent change: one objective, an exit code that CI can gate on, and a sealed evidence pack.

npm install -g @testmuai/kane-cli && kane-cli login

Conclusion

Add the repeat-run rule to your CLAUDE.md or AGENTS.md this week, then turn on --fail-on-flaky-tests in CI so the rule has a gate behind it.

AI test writing helps only when the tests it adds can be trusted. Deterministic testing keeps test flakiness out of the suite, and a separate verifier keeps the pass or fail call away from the agent that wrote the code.

Testing from the terminal? Kane CLI by TestMu AI lets SDETs and developers plan, write, run and debug tests from the command line.

To run your first test, start free, or

book a demo

for a full walkthrough of the product.