A Practical Quality Gate for Modern Web Apps: From AI-Built Pull Requests to Reliable E2E Coverage

What a Quality Gate Actually Is

A condition that must hold before a change moves forward. Not a dashboard, not a report, and not a convention people agree to follow. Something that stops the merge.

On GitHub the mechanism is a required status check on a protected branch. GitHub distinguishes two configurations, and the difference decides whether the check ever ran against the code you are actually merging.

• Strict - the Require branches to be up to date before merging box is ticked, so the branch must be current with its base before the merge is allowed.
• Loose - that box is unticked, so a check can pass against a branch that has since diverged from main.
• Reviews are a separate control - requiring a number of approving reviews is its own setting, and it does not imply anything ran.

Loose checks are the reason a green pull request can break main on merge. Nothing lied. The check simply answered a question about a different tree.

Why AI Pull Requests Break the Old Gate

The old gate had an unwritten fourth layer: a person who read the diff and noticed things. That layer had a throughput limit nobody had to state, because human authorship enforced it.

Agent authorship removes the limit on one side of the equation and not the other.

• Volume outruns attention - reviewers skim, and skimming finds style problems rather than logic problems.
• Plausibility is no longer a signal - agent-written code reads well by default, so looking reasonable stopped narrowing anything down.
• The tests share the author - when one process wrote the feature and the suite, a green result confirms an interpretation rather than checking it.
• The description is generated too - a pull request body describing what was intended is not evidence of what was built.

The third point is the structural one, and we unpacked it in whether coding agents can test their own code.

What the Gate Has to Prove

Four things, in order of how expensive they are to get wrong. Each question is answerable by something other than the process that wrote the code.

Row three is the one most pipelines are missing entirely. It is also the row where agent-written changes fail most visibly, because nothing in the source says a button stopped being clickable.

The Four Layers

Order them by how fast they fail, so the cheapest signal arrives first and nobody waits four minutes to learn about a missing import.

• Static analysis, seconds, blocking. Types, lint, and formatting, with formatting fixed automatically rather than reported.
• Unit and integration tests, a minute or two, blocking. Keep the assertions the agent did not author under stricter review than the ones it did.
• Browser verification, under a minute per flow, blocking on the flows you cannot afford to break.
• Human approval, scoped narrowly to payments, authentication, data deletion, and migrations rather than applied to everything.

Layer four is where teams go wrong in both directions. Approve everything and the agent stops being useful, approve nothing and the first bad merge is expensive.

Wiring the Gate in CI

Layer three is the one that needs a tool most pipelines do not already have. It has to run headless, decide without a test file, and fail the job on its own exit status.

Kane CLI from TestMu AI meets those three constraints, and its exit codes are deliberately distinct so a blocked environment never reports as a broken product.

- name: Browser verification
  run: |
    npm run build
    npm run preview &
    sleep 5

    kane-cli run --agent --headless \
      "sign in as the demo user, add an item to the cart,
       complete checkout, and assert the confirmation page shows an order number" \
      --url http://localhost:4173/

# exit 0 verified   exit 1 assertion failed   exit 2 environment   exit 3 timeout

- name: Publish the evidence pack
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: browser-evidence
    path: .testmuai/evidence/

The second step matters as much as the first. A verdict the reviewer cannot open is a request for trust, and per-step screenshots with a network log and console output turn it into something checkable in ninety seconds.

The always condition is deliberate too, because the evidence from a failed run is the evidence you most want. Full setup steps are in connecting Kane CLI to GitHub Actions.

The Skipped Check Trap

A job that skips satisfies a required status check. That is documented behaviour, not a bug, and it is how a carefully built gate quietly stops gating anything at all.

The usual cause is a path filter written months earlier. Somebody restricts the browser job to changes under a directory, the application moves, and the job skips on every pull request while the branch stays green.

• Assert the job ran - have it write a marker artifact and add a small check that fails when the marker is absent.
• Watch the run count, not the pass rate - a gate that stops firing looks identical to a gate that keeps passing.
• Prefer a conditional inside the job - a job that always runs and exits early is visible, and a job that never starts is not.
• Re-read path filters when the repository moves - a directory rename disarms every filter that mentioned the old name.

The same failure mode applies to any timeout-based control, including agent lifecycle hooks, which is covered alongside the other limits in Claude Code hooks.

Who Can Bypass the Gate

By default, more people than you think. GitHub documents that branch protection restrictions do not apply to people with admin permissions on the repository, or to custom roles carrying the bypass branch protections permission.

Administrators can turn that off and apply the restrictions to everyone, and on a repository where an agent opens pull requests, that setting is worth revisiting.

• Audit who holds admin - the bypass list is usually longer than the list of people who need it.
• Decide about automation accounts explicitly - a bot with admin rights inherits the bypass without anyone deciding it should.
• Log the overrides you keep - an emergency bypass is reasonable, an unrecorded one is not.

What to Build First

Do not build all four layers in a sprint. Build the one that would have caught the last thing that broke.

• Pick the single flow whose failure would be worst, and write one browser objective for it.
• Make that check required on the protected branch, with the strict setting so it runs against the merged result.
• Publish its evidence as a build artifact on every run, including failures.
• Add the marker assertion that proves the job ran, then add the second flow.

Step four before step five is the discipline that keeps this working a year later. A gate nobody monitors becomes a gate nobody notices has opened.

For the same idea extended past the merge and into production, see continuous verification for AI-generated code. And for the review-time half of the problem, AI code review versus verification separates what reading a diff can and cannot establish.