E2E Tests in Pull Requests: Build a Merge Gate Teams Trust

What Does It Mean to Run E2E Tests in Pull Requests?

A pull request gate is a CI job triggered by the pull_request event that runs a selected set of end-to-end testing scenarios against the proposed change, then reports back as a status check on the pull request. When that check is marked required in branch protection, a red result blocks the merge button.

The distinction that matters is where validation happens relative to review. Running the same suite after merge tells you the main branch is broken; running it on the pull request tells the author while they still have the code in their head.

Industry data suggests the post-merge model is losing ground. CircleCI's 2026 State of Software Delivery reports that feature branch throughput rose 15% year over year while main branch throughput fell 7%, meaning more verification work is shifting onto branches before they land.

Three properties separate a gate that lasts from one that gets removed:

• Speed - it returns before the reviewer moves to other work, which is a shorter window than most pipelines assume.
• Trust - a red result reliably means a real defect, so nobody develops the habit of re-running until green.
• Legibility - the failure explains itself where the reviewer already is, rather than in a log file three clicks away.

Most published advice covers the first property and treats the other two as tooling details. In practice, trust and legibility are what determine whether the gate is still required six months later.

Why Do PR Test Gates Get Switched Off?

Gates are rarely removed in a single decision. They decay through a sequence that is recognisable across teams, and each step is individually reasonable.

• A test fails for a reason unrelated to the change, and the author re-runs it.
• Re-running becomes the first response to any red check, before anyone reads the failure.
• Someone adds a blanket retry so the re-run happens automatically.
• The gate now reports green for both passing tests and tests that failed once, and the signal is gone.
• A release is delayed by a red check nobody trusts, and the check is demoted to non-blocking.

The failure rate that drives this loop is rising rather than falling. Bitrise's Mobile Insights report, published in November 2025 and drawn from more than 10 million builds between January 2022 and June 2025, found the proportion of teams experiencing any test flakiness grew from 10% to 26%.

What that erosion costs downstream is visible in main branch health. CircleCI's 2026 data puts main branch success rates at 70.8%, the lowest in more than five years and well under its recommended benchmark of 90%, with the typical team taking 72 minutes to get back to green against a 60-minute benchmark.

A gate that is bypassed does not fail quietly. It relocates the failure to the branch everyone shares, where recovery is measured in hours rather than in the minutes a pull request run would have taken.

How Long Should a PR Gate Take?

Published guidance tends to assert a number, usually five or ten minutes, without showing where it comes from. A borrowed number is not useful, because the same ten minutes is trivial for a team merging four pull requests a week and unaffordable for one merging forty a day.

Derive your own budget from three figures you already have:

• Merged pull requests per working day, taken from your repository insights over the last month.
• Runs per pull request, which is higher than one because review comments produce new commits. Two to three is typical.
• Waiting cost per run, counted only for the runs where the author actually waits rather than switching tasks.

Multiply the three and the answer arrives in engineer-hours per week. A team merging 20 pull requests a day at 2.5 runs each spends roughly 4 engineer-hours per week for every minute the gate takes, which makes the difference between a 6-minute and a 16-minute gate about 40 hours weekly.

That calculation reframes the question. The budget is not a best practice to adopt; it is the point where the waiting cost exceeds what catching a defect early is worth to you, and it moves as your merge rate moves.

When the budget is tighter than the suite, the first lever is parallelism rather than deletion. Splitting work across workers and shards is covered in depth in this guide to Playwright parallel testing with workers and sharding, and the same arithmetic applies to any framework.

Which E2E Tests Belong in a Pull Request?

This is the one question where credible sources openly disagree, and the disagreement is worth stating plainly because most articles present only one side.

• The subset position - run a smoke selection on the pull request and the full suite after merge, keeping the gate inside its time budget.
• The trap position - a subset gives false confidence, because the tests you cut are exactly the ones nobody watches until an incident.
• The informational position - browser tests are too unstable to block a merge at all and belong as advisory checks alongside required lint and unit runs.

All three are defensible, and which one applies to you depends on a property you can measure rather than a preference. Use your suite's flake rate as the decision rule.

Selecting the subset by user-visible risk beats selecting it by folder or by tag inherited from a previous team. Authentication, checkout, and any path that moves money or data earn a slot ahead of a test that asserts a settings label. The practice of smoke testing in a CI/CD pipeline gives a reasonable starting shape for that selection.

One category deserves a specific mention because functional assertions miss it entirely. A change can pass every interaction test and still ship a broken layout, which is what visual regression testing is designed to catch on a diff.

How Do You Stop Flaky Tests From Voiding the Gate?

Retries are where most gates quietly lose their meaning. A test that fails and then passes on attempt two renders in the pull request as a green check, visually identical to a test that passed the first time, so the reviewer cannot tell the difference and neither can the merge button.

The fix is to make retries conditional on the kind of failure rather than on the fact of failure. Infrastructure signatures such as a dropped session or a resolver timeout are worth absorbing; an assertion that says the total was wrong is never worth retrying.

TestMu AI's HyperExecute test orchestration cloud implements that distinction directly in configuration. Retries can be scoped to matching error patterns, so a known infrastructure exception is retried while a failed assertion fails immediately:

retryOnFailure: true
maxRetries: 3
retryOptions:
  errorRegexps: ["org.openqa.selenium.NoSuchElementException"]

failFast:
  maxNumberOfTests: 2
  level: scenario

The failFast block in the same file addresses the opposite waste. When a change breaks something fundamental, the job aborts after the configured number of consecutive failures instead of burning the full gate budget proving the same breakage twenty more times, and the counter resets on any pass so intermittent flakiness does not trigger it.

Two further HyperExecute behaviours shorten the feedback loop on a pull request. Previously failing tests are automatically reordered to run first, so a reintroduced defect surfaces early rather than at minute nine, and dynamicAllocation hands test cases to workers at runtime so a fast machine pulls more work instead of idling while a slower one finishes.

Configuration alone will not fix a suite that is genuinely unstable, and the underlying causes are worth understanding before tuning retries. This reference on flaky tests and how to detect them covers the detection side in detail.

What Should a Reviewer See When a UI Test Fails?

Almost every guide on this topic asserts that failures should be actionable, and almost none of them show what that looks like in the pull request. The gap matters because the default GitHub experience for a failed end-to-end job is a check name, a red cross, and a link to a log.

A failure is legible when the reviewer can answer three questions without leaving the pull request:

• Which behaviour broke - stated as the user-facing step that failed, not as the name of a test method.
• Whether this change caused it - distinguishing a regression from a pre-existing failure or an environment fault.
• What to do next - a concrete first action rather than an invitation to read four hundred log lines.

TestMu AI's GitHub App integration is built around that constraint. It posts a comment that updates as the run progresses, lists the test cases that executed with links into Test Manager, and finishes with a validation report carrying pass and fail counts, individual failure detail, root cause analysis, and an explicit approve, request-changes, or investigate recommendation.

The root cause layer is what turns a log into a diagnosis. It separates the primary cause from the cascading symptoms that follow it, labels the failure category, and returns numbered remediation steps, with manual analysis documented to complete in 20 to 30 seconds.

Triage can also be scoped so the reviewer is not buried in noise. Automatic analysis can be limited to new failures, defined as a test that failed after passing at least 10 consecutive times, or to consistent failures that failed in all of the previous 5 runs, which are the two categories most likely to represent a real regression in the diff under review.

None of this replaces human judgement about the change itself. It removes the tab-switching tax that makes reviewers stop reading red checks, which is the specific behaviour that kills gates. The wider practice around that review loop is covered in this guide to the code review process.

How Do You Gate a Pull Request on Real Devices?

Web gates run headless browsers on hosted Linux runners, which is appropriate for a web application and silently inadequate for a mobile one. A hosted runner never encounters a notch cutting into a fixed header, a gesture conflict with the system back navigation, or the memory pressure that only appears on a mid-range handset.

Gating a mobile pull request means the CI job builds the artifact from the branch, uploads it, and runs the selected scenarios across a device matrix before the check reports back. TestMu AI's real device cloud provides 10,000+ real Android and iOS devices for that step, which removes the physical device lab from the critical path of a merge.

Three constraints apply to a device gate that do not apply to a browser gate:

• Build time counts against the budget - compiling the app is part of the gate, so a five-minute build leaves five minutes of a ten-minute budget for tests.
• Device selection is a risk decision - two or three devices chosen from your own analytics beat a broad matrix that nobody can wait for.
• Queue wait is real latency - concurrency limits mean a busy afternoon can add minutes before the first test starts, so the budget has to be measured end to end rather than from first test to last.

The mobile case is also where flakiness data is most sobering. The same Bitrise research found that teams using monitoring tools experience 25% fewer flaky reruns, which is a direct argument for instrumenting the gate rather than only building it.

What Breaks When You Make the Gate Required?

Promoting a green job to a required status check exposes several mechanics that setup guides skip, and each one has produced a broken gate in a real repository.

• Sharded jobs create multiple check names - a four-way matrix reports four separately named check runs, and every one of them has to be marked required. Changing the shard count renames them and silently drops the protection.
• Fork pull requests receive no secrets - GitHub's documentation states that "with the exception of GITHUB_TOKEN, secrets are not passed to the runner when a workflow is triggered from a forked repository". A gate needing a grid key or device-cloud token therefore cannot authenticate on an outside contribution.
• A skipped job is not a passed job - if a path filter causes the workflow not to run, a required check can sit pending forever and block the merge rather than waving it through.
• Draft pull requests do not need the full gate - excluding drafts and documentation-only diffs is the cheapest speed win available, and it costs nothing in coverage.

The fork constraint is documented in GitHub's guide to using secrets in GitHub Actions. For an open-source repository, the workable pattern is an advisory run on fork pull requests and a full authenticated gate once a maintainer has reviewed the diff.

Wiring the execution itself is deliberately uniform. The pattern is to download the CLI in the job, supply LT_USERNAME and LT_ACCESS_KEY from repository secrets, and invoke the binary against a configuration file:

name: E2E Gate
on:
  pull_request:
    types: [opened, synchronize, reopened]
    paths-ignore:
      - '**/*.md'
      - 'docs/**'

jobs:
  e2e:
    if: github.event.pull_request.draft == false
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run gated E2E suite
        env:
          LT_USERNAME: ${{ secrets.LT_USERNAME }}
          LT_ACCESS_KEY: ${{ secrets.LT_ACCESS_KEY }}
        run: |
          wget https://downloads.lambdatest.com/hyperexecute/linux/hyperexecute
          chmod u+x hyperexecute
          ./hyperexecute --config pr-gate.yaml

The same three steps apply to GitLab, CircleCI, Jenkins, Azure DevOps, and Bitbucket, because any system that can run a CLI command can run the gate. Setup details are in the HyperExecute documentation, and a framework-specific walkthrough is available for running Cypress tests with GitHub Actions.

How Do You Roll the Gate Out Without Blocking Everyone?

A gate that becomes required on its first day will fail on someone's unrelated pull request within the week, and that first bad experience sets the team's opinion of it. Earning the required flag over four weeks avoids that.

• Week one, run it non-blocking - trigger the job on every pull request with the check advisory. Nobody is blocked and you begin collecting real data instead of estimates.
• Week two, measure the flake rate - re-run the identical commit on a schedule. Any variation in result with no code change is flake, and this number decides which row of the selection table you are in.
• Week three, fix or quarantine - repair the top offenders and quarantine the rest with a named owner and a fix-or-delete date. A quarantine with no expiry becomes a permanent graveyard.
• Week four, promote what earned it - mark the stable subset required, leave the rest advisory, and confirm every shard name is listed in branch protection.

Publish the flake number where the team can see it. A gate that is trusted because everyone has seen its measured reliability behaves differently from one that is trusted because a lead asserted it.

If your suite does not exist yet, the sequencing question changes. The related discussion of E2E test coverage on every pull request covers building that coverage, and the broader mechanics of integrating E2E tests into your CI/CD pipeline apply once the gate is stable.

Where to Start

Open your repository insights, count the pull requests merged in the last thirty days, and divide by working days. Multiply that by 2.5 runs and by the minutes your current suite takes, and you have the weekly engineer-hours your gate costs today. That single number tells you whether your problem is speed, selection, or trust before you change any configuration.

Then run the gate non-blocking for a week and measure its flake rate on an unchanged commit. The selection table above maps that rate to a decision, and running E2E tests in pull requests only pays off once the rate is low enough for the row you land on.

For the execution side, HyperExecute runs suites up to 70% faster than traditional grids by keeping the test script and its execution components in one isolated environment, and it carries the retry scoping, fail-fast, and failure reordering controls a gate depends on. Pair it with the GitHub App so failures arrive as a diagnosis inside the pull request rather than as a link the reviewer will not open.