Continuous Testing Strategy: How to Build One With AI Agents

What Is a Continuous Testing Strategy?

A continuous testing strategy is the pipeline-level plan behind your CI configuration. It is narrower than a project-wide test strategy, which covers scope and approach for a whole product, and it explains each gate in the pipeline file: why a check blocks the merge, why another runs nightly, and who changes the rule when it stops working.

DORA's continuous integration guidance sets the baseline: every commit triggers a build, and a set of automated tests runs both before and after the merge to catch regression bugs.

On top of that baseline, a usable strategy records these decisions:

• Stage map - which tests run on a pull request, after merge, nightly, before a release, and in production.
• Gate rules - which failures block a merge or a deployment, and which only open a ticket for the owning team.
• Time budgets - how long each blocking stage may take, measured from push to result and including machine setup.
• Environments and data - where each stage runs, which dependencies are virtualized, and how test data is created and cleaned up.
• Flake policy - which failures earn an automatic retry, when a test is quarantined, and who fixes it by when.
• Decision owners - which calls a deterministic check makes, which an AI agent may make, and which need a person.

Which Tests Should Run at Each Pipeline Stage?

Put the fastest and most deterministic checks closest to the commit, and move slower, broader, or noisier checks later, where they inform a decision instead of holding every merge.

Gating stages need environments that behave the same on every run. Service virtualization stands in for third-party APIs and unfinished services, so a pull request does not fail because a partner sandbox is down. Seed test data per run so one job's leftover cart or account cannot change another job's result.

Moving checks toward the commit is the shift left testing half of the plan, and production monitoring is the shift-right half. To pick tools for each row, compare the options in this roundup of continuous testing tools.

Note: Run the pull request gate and the nightly matrix on TestMu AI, with 3,000+ browser/OS combinations and 10,000+ real devices in one cloud. Start testing for free

Where Do AI Agents Fit in a Continuous Testing Strategy?

Give agents the work that produces candidates and evidence: drafting tests, healing locators, and exploring flows. Keep every decision that can stop or ship a change, including which new tests join the gate, with deterministic checks and named people.

GitHub's Octoverse 2025 report counted 1+ million pull requests created by its Copilot coding agent between May and September 2025.

Each of those pull requests goes through the same gates as human code. A strategy written before agents arrived assumes a person wrote the tests and a person reads the results, so it needs explicit rules for the jobs agents now do:

• Drafting tests from a change - after a @TestMuAI Validate this PR comment, the KaneAI GitHub App reads the pull request's diff, description, and README, generates end-to-end tests, runs them in parallel, and posts results with a root cause analysis back to the pull request. Keep generated tests advisory until a reviewer checks their assertions against the requirement; this walkthrough of pull request validation with the KaneAI GitHub App shows the flow end to end.
• Healing broken locators - self-healing re-anchors a step when the UI changes and surfaces the change for review. Approve each heal before it becomes permanent, because aggressive healing can hide an element that really disappeared.
• Exploring flows - Kane CLI runs a plain-English objective in a headless browser and exits with 0 for a pass, 1 for a failed assertion, 2 for an error, and 3 for a timeout, so a nightly job can act on its result without parsing logs.

KaneAI, TestMu AI's GenAI-native testing agent, is built around this split: you review and approve a generated test plan before it runs, you can pause a run to correct a step, and your input takes priority over the agent's decisions. For how agent-driven QA works beyond the pipeline, see this guide to agentic QA.

What Does a Pull Request Gate Look Like in Practice?

To make the pull request stage concrete, I wrote a six-test smoke suite in pytest and Selenium for TestMu AI's Ecommerce Playground. It checks search results, the empty-search message, a product page's price and Add to Cart button, the add-to-cart confirmation, a rejected login, and the link to registration. Each test opens a cloud browser session, reads credentials from the LT_USERNAME and LT_ACCESS_KEY environment variables, and reports pass or fail back to the session.

I ran the suite on HyperExecute, TestMu AI's test orchestration cloud, with HyperExecute CLI 0.2.358. The two blocks below are the hyperexecute.yaml that ran, split in half so each block stays readable; the first half sets the gate policy:

# hyperexecute.yaml, part 1 of 2: gate policy
version: 0.1
runson: win
autosplit: true
concurrency: 3
globalTimeout: 30
jobLabel: ["pr-gate", "high"]

retryOnFailure: true
maxRetries: 2
retryOptions:
  errorRegexps: ["ERR_CONNECTION", "ERR_NAME_NOT_RESOLVED", "session not created"]

• autosplit and concurrency - HyperExecute discovers the test files and spreads them across three virtual machines, one file each.
• retryOnFailure and retryOptions - a failed file is retried up to twice, but only when its log matches one of the listed infrastructure errors: a dropped connection, a DNS failure, or a browser session that never started.
• globalTimeout and jobLabel - the whole job is capped at 30 minutes, and the "high" label puts it ahead of medium and low priority jobs when concurrency is contended.

The second half installs the pinned dependencies (Selenium 4.50.0 and pytest 9.1.1) and tells HyperExecute how to find and run the tests:

# hyperexecute.yaml, part 2 of 2: runtime and test discovery
runtime:
  language: python
  version: "3"
pre:
  - pip install -r requirements.txt

testDiscovery:
  type: raw
  mode: remote
  command: grep -rl def.test_ tests
testRunnerCommand: pytest -v $test

Running the CLI against that file dispatched job 80e232af, which printed this stage timeline (copied from the terminal as shown, without the upload lines before it and the summary after it):

✔ [1]  setup-runtime (4s)
✔ [1]  pre (14s)
✔ [1]  discovery (0s)
✔ [2]  setup-runtime (4s)
✔ [3]  setup-runtime (5s)
✔ [4]  setup-runtime (5s)
✔ [2]  pre (13s)
✔ [3]  pre (13s)
✔ [4]  pre (14s)
✔ [3]  tests/test_search.py (23s)
✔ [2]  tests/test_product.py (24s)
✔ [4]  tests/test_account.py (24s)
 COMPLETED

The three test files ran in parallel and finished in 23 to 24 seconds each. The job summary reported a job duration of 1m50s, with four tasks (the discovery task plus three test machines) and zero failed tasks.

That is about the same as the 108 seconds the same six tests took when I ran them one after another from my laptop against the same cloud grid. Every machine spent 17 to 19 seconds setting up Python and installing requirements, and discovery ran on its own machine first, so machine setup and orchestration, not the tests, took most of the wall-clock time for a suite this small.

Measure the gate end to end and cache dependencies before adding concurrency; the HyperExecute YAML parameters reference documents the cacheKey and cacheDirectories keys for this. Parallel machines pay off once the suite is larger than one machine can finish inside the budget.

How Do You Keep Flaky Tests From Eroding the Gate?

Decide in advance which failures earn an automatic retry, where a flaky test goes after it keeps failing for reasons unrelated to the code, and what it takes to come back. A gate that retries everything hides real defects, and a gate that never retries trains developers to rerun the pipeline by hand.

• Retry only infrastructure errors - match retries to error signatures such as connection resets and session-creation failures, and let assertion failures fail on the first attempt.
• Quarantine with an owner - move a repeat offender out of the gating suite, assign it to a team with a due date, and keep running it outside the gate so its health stays visible.
• Mute visibly - HyperExecute lets you mute a test scenario from a job's results so later runs skip it, and the scenario stays inactive until someone unmutes it, which keeps every muted test on a visible list.
• Watch failure frequency - Test Insights ranks tests by how often they fail across runs, and its AI root cause analysis correlates network, console, and framework logs into a likely cause that the owner confirms before acting.

Atlassian's write-up of Flakinator, its tool for managing flaky tests, reports that as of its last quarter the tool had recovered more than 22,000 builds and identified 7,000 unique flaky tests.

The process behind those numbers is easy to copy: Flakinator files a Jira ticket for the owning team with a pre-decided due date, and a test leaves quarantine only after it stays healthy for a configured period.

To confirm the retry rule in my YAML behaves as written, I ran a copy of the suite with two deliberate faults. The product test expected the title "iPhone 17" instead of "iPhone", and the first search test raised a WebDriverException carrying net::ERR_CONNECTION_RESET on its first attempt only. With the same hyperexecute.yaml, job 28492d40 printed these test stages:

x [2]  tests/test_search.py (22s)
✔ [4]  tests/test_account.py (24s)
x [3]  tests/test_product.py (25s)
✔ [2]  {retry 1} tests/test_search.py (11s)

The assertion failure ran once and was not retried, because no pattern in errorRegexps matched it. The injected connection reset matched ERR_CONNECTION, so HyperExecute retried the file and the retry passed. The job summary then reported FAILED with one failed task, and the CLI exited with code 1 after printing "Error: Failed tasks found.", which is what fails the CI step and blocks the merge.

The retry reran the whole file, both tests included, because the unit of retry is the runner command, so keep gating test files small enough that a retry costs seconds.

I left fail-fast out of this gate after testing it. With a plain pytest runner, failFast at its default test level never fired, even when three test files failed in a row on one machine; with level: scenario, which counts each test file as one scenario, it stopped the job after the second failing file. In the two-fault run, scenario-level fail-fast also counted the search file's first, retryable failure, cancelled the retry, and stopped the account file mid-run, so keep it for suites that queue more files than machines.

Austin Siewert

Co-Founder, Steadfast Systems

Discovered @TestMu AI yesterday. Best browser testing tool I've found for my use case. Great pricing model for the limited testing I do 👏

2M+ Devs and QAs rely on TestMu AI

Deliver immersive digital experiences with Next-Generation Mobile Apps and Cross Browser Testing Cloud

Sign up with googleSign up with Email

How Do You Continuously Test an Agentic AI Application?

When the product itself is an AI agent, the same input can produce a different but acceptable output on the next run, so a single green run proves little. The strategy changes in three places: how often a scenario runs, where each evaluation sits in the pipeline, and what the gate reads.

Red Hat's engineers state the first point plainly in their write-up on CI/CD for agentic AI: "One test run is never enough. Responses can change with every run, so you need multiple tests to ensure the system works as expected."

The same Red Hat team found that pull request checks alone could miss intermittent failures, and relied on nightly runs that evaluate more generated conversations and configurations before trusting a prompt change or a dependency upgrade.

• Repeat each scenario - judge a pass rate across several runs, and decide whether the feature needs one success or every success; this comparison of pass@k vs pass^k covers when each metric fits.
• Split evaluations by stage - keep a small, fast evaluation on pull requests, move the long evaluation to the nightly run, and budget model spend per run with LLM cost tracking for agent evals.
• Gate on evidence - for agents that call tools, write files, or open pull requests, TestMu AI's Agent Assurance runs from the terminal as rook, checks each criterion against what the run changed, and reports separately what it could not verify. Its Agent Assurance CI/CD guide warns that a zero exit code is not an agent-quality gate and keeps model-backed scenario generation outside the release gate. The product is pre-alpha, so expect its command surface to change.
• Grade conversations separately - for agents that talk, such as chat, voice, and phone agents, Agent Testing grades the replies across its quality metrics.

The full release loop for agents, from pre-merge checks to production feedback, is covered in this guide to continuous AI agent testing. In the TestMu 2026 talk below, Pooja Oza covers how agentic systems break the same-input, same-output assumption behind pass/fail testing, and where human judgment stays in the loop.

How Do You Know Your Continuous Testing Strategy Works?

Review a short set of numbers every month and move tests between stages based on them:

• Gate time - push to result for the pull request stage, including machine setup, compared against the budget you set.
• Flake rate of gating tests - how often a gating test fails and then passes on the same commit; a rising rate means the gate is losing trust.
• Quarantine age - how long tests stay out of the gate, since every quarantined test is behavior nobody is checking on merge.
• Escaped defects and change failure rate - bugs found in production and the share of deployments that need a fix; change failure rate is one of the DORA metrics.
• Unverified share for agents - for agentic features, the share of criteria each run could not verify, which should shrink as the agent records more of what it does.

Conclusion

Start with the pull request gate: list every check that can block a merge, time it from push to result including setup, and move anything that misses your budget to post-merge or nightly. Then put one agent in the evidence lane by installing the KaneAI GitHub App, and follow the GitHub App integration guide to set the config file and trigger flags.