Green CI Isn't Proof: Continuous Verification for AI Code

What Is Continuous Verification for AI-Generated Code?

Continuous verification is the practice of proving that each change behaves as intended before it merges, by exercising the running application rather than inspecting the code that produced it. It answers a narrower question than a test suite does: not whether old assertions still hold, but whether this change does what it was asked to do.

All three practices below run in a pipeline and report pass or fail, which is why they get conflated.

None of the three replaces the others. Verification is the layer that moves with the change instead of trailing it, which is the same reasoning behind shift left testing, applied at the point of merge rather than the start of a sprint.

Why Do Green Pipelines Still Ship Broken Behavior?

Because a pipeline reports on the assertions it was given, and generated code is very good at satisfying the checks a compiler and a linter make. The Veracode update quantifies that split across 80 coding tasks in four languages, and the headline number has barely moved since 2023.

Where the gap concentrates matters more than the average. Two vulnerability classes in that study sit far below the overall pass rate while two others are largely handled, which is why a single aggregate figure hides the risk.

• Secure generations are a coin flip - only 55% of generation tasks in the Veracode study produced secure code, so 45% introduced a known flaw while the syntax looked clean.
• Compilation is not correctness - the same data puts syntax pass rates above 95% against security pass rates near 55%, so the signal a build gives you has decoupled from the signal you need.
• Language choice moves the odds - Veracode reports Java at a 29% security pass rate against Python at 62%, so an identical review process yields very different residual risk per service.
• Two classes stay broken - cross-site scripting reached a 15% security pass rate and log injection 13% in that study, while SQL injection sat at 82% and insecure cryptographic algorithms at 86%.
• The suite is a record of past thinking - every assertion in it was written for a behavior someone had already imagined. A change that introduces a behavior nobody imagined passes cleanly.
• Static analysis reads the code as written, which means a flaw expressed as a legitimate-looking control flow reads as intentional rather than as a defect.

The practical consequence is that green stopped being a decision-grade signal for agent-authored changes. It still tells you the build is sound, which is worth having, but it no longer tells you the feature works.

What Changed About the Defects Themselves?

The defect population shifted rather than shrank, and it shifted toward the class that only runtime behavior exposes. Apiiro's analysis of AI-assisted development, run with its Deep Code Analysis engine across tens of thousands of repositories, found trivial syntax errors in AI-written code down 76% and logic bugs down more than 60%.

The movement in deeper flaws went the other way. Those are broken auth flows and systemic design weaknesses, the kind that compile perfectly and only show themselves when a real user session walks through them.

Human code review capacity moves the wrong way at the same time. Apiiro reports that AI-assisted developers produced 3 to 4 times more commits than their non-AI peers, but packaged them into fewer pull requests, each significantly larger in scope and touching more files.

• Cheap defects got cheaper to catch - typos and null dereferences were already handled by the compiler and the linter, so removing them changes very little about your risk.
• Expensive defects got more common - Apiiro recorded privilege escalation paths up 322% and architectural design flaws up 153%, and an authorization path that grants more than it should is invisible to every gate that reads source rather than exercising the application.
• The human gate degrades under load - a reviewer reading a 900-line diff catches less per line than one reading 90, and the diffs got bigger exactly as the volume rose.
• Larger merges also widen the blast radius per merge, so each unverified change carries more of the release with it.

This is the argument for behavior-level gates rather than more static ones. The same pressure is reshaping how teams scope agentic regression testing, where the question is which checks a machine can own and which ones a human still has to sign.

Who Verifies the Tests the Agent Wrote?

This is the question most continuous verification write-ups skip, and it decides whether the rest of the pipeline means anything. If the agent that wrote the code also wrote the tests from that same code, a bug in the implementation gets copied into the assertion that was supposed to catch it.

There is measurement behind that concern. A 2026 study on the misguidance effect in LLM-generated unit tests found that prompting a model with buggy code steers it toward tests that validate the erroneous behavior rather than expose it, and that the effect is twofold: misguided tests go up while effective, bug-finding tests are suppressed.

The authors' fix is the useful part for pipeline design. Replacing the code under test in the prompt with a generated behavioral specification reduced misguided tests and increased effective ones, which is the same principle as writing your acceptance criteria before you look at the diff.

Three rules keep a verifier independent enough to be worth running:

• Derive checks from intent, not from the diff - write the objective from the ticket, the acceptance criteria, or the user story. If the only input to your check was the code, the check cannot disagree with the code.
• Assert on rendered behavior, not internal functions - a check that calls the same function the agent wrote shares its assumptions. A check that loads the page and looks at what a user would see does not.
• Keep at least one check outside the agent's write access - if the agent can edit the assertion that blocks its own merge, a failing run is one file edit away from passing.

None of this means agent-written tests are worthless. They are good at breadth and cheap to produce, so let them cover the wide surface, and let an independent behavioral check own the merge decision.

The failure mode has its own depth, and whether coding agents can test their own code works through exactly where the shared interpretation between code and test starts to bite. The rest of this article takes independence as settled and deals with the pipeline side: where the check runs, and what it has to hand back. The same separation drives verifying vibe-coded software, where the code arrives faster than anyone can read it.

Where Does Verification Belong in the Pipeline?

In four places, with different budgets. The common failure is putting every check at PR time, which turns verification into the slowest step and gets it switched off within a month.

The stage that earns its keep first is the agent loop, because a defect caught there never consumes review attention at all. Once the loop and the PR gate are running, the post-merge and production stages are largely a scheduling problem, which is the shape of continuous agent testing as a full lifecycle rather than a single gate.

How Do You Gate a Merge on Behavior?

You need three things the pipeline can rely on: a browser the check can drive, an objective written from intent, and an exit code the workflow can branch on. Kane CLI from TestMu AI is built for that shape, running the same binary headed on a laptop and headless on a CI runner, driving real Chrome through the DevTools Protocol and taking the objective in plain English rather than as selectors.

The exit codes are what make it a gate rather than a report: 0 for passed, 1 for failed, 2 for error, 3 for a timeout or cancellation. A required status check that distinguishes a real failure from a runner problem is the difference between a gate people trust and one they override.

name: Verify PR behavior
on: [pull_request]

jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: browser-actions/setup-chrome@v1
      - run: npm install -g @testmuai/kane-cli
      - name: Verify checkout still completes
        env:
          LT_USERNAME: ${{ secrets.LT_USERNAME }}
          LT_ACCESS_KEY: ${{ secrets.LT_ACCESS_KEY }}
        run: |
          kane-cli run "Add a product to the cart and complete checkout" \
            --url https://staging.myapp.com \
            --headless --timeout 300 \
            --username "$LT_USERNAME" \
            --access-key "$LT_ACCESS_KEY"

Two flags in there are not optional in CI. A runner has no display, so --headless is required, and a hung run will hold the queue indefinitely without --timeout. The Kane CLI documentation covers authentication with CI secrets and the remote browser endpoint for runner images that cannot install Chrome.

A verdict is only useful if it carries evidence. The check below is one we ran on TestMu AI cloud while writing this article, against the public ecommerce playground, collecting console errors and failed responses alongside the functional assertion.

import { Browser } from '@testmuai/browser-cloud';

const client = new Browser();
const session = await client.sessions.create({
  adapter: 'playwright',
  lambdatestOptions: {
    browserName: 'Chrome',
    browserVersion: 'latest',
    'LT:Options': { build: 'Continuous Verification', name: 'Search returns results' }
  }
});
const { browser, page } = await client.playwright.connect(session);

const consoleErrors = [], netFailures = [];
page.on('console', m => { if (m.type() === 'error') consoleErrors.push(m.text()); });
page.on('response', r => { if (r.status() >= 400) netFailures.push(r.status() + ' ' + r.url()); });

await page.goto('https://ecommerce-playground.lambdatest.io/');
await page.fill('input[name="search"]', 'iphone');
await page.press('input[name="search"]', 'Enter');
await page.waitForSelector('.product-thumb');

console.log('results', await page.locator('.product-thumb').count());
console.log('console_errors', consoleErrors.length);
console.log('http_failures', netFailures.length);

The run completed in 5.2 seconds on a cloud Chrome session and produced this output, recorded under build 102530643:

results 4
console_errors 0
http_failures 0

Three numbers are a defensible verdict. A reviewer can see that four results rendered, that nothing threw in the console, and that no request came back 4xx or 5xx, without rerunning anything or trusting a checkmark.

How Do You Know the Loop Is Working?

Pipeline pass rate is the wrong metric here, because a gate that never fails is indistinguishable from one that is not running. Four measures tell you whether verification is changing outcomes.

• Escaped defects on agent-authored changes - track this separately from your overall escape rate. If the two diverge, you know which population of changes your process is failing.
• Coverage of changed flows - the share of merged pull requests that touched a user-facing flow and had a behavioral check on it. This is the number that should climb first.
• Time from merge to first failing signal - if production is where you learn, the gate is in the wrong place. Falling values mean detection is moving left.
• Verification flake rate - a check that fails for environmental reasons trains people to override the gate. Fix flaky tests before adding new checks, or the gate loses its authority.

Reading those trends across hundreds of runs is a data problem rather than a testing one. TestMu AI's test intelligence surfaces failure patterns and flaky-test signals across builds, which is what turns four metrics into a decision about where the next check should go.

Where to Start This Week

Pick the single flow whose breakage would page someone, write one objective for it from the acceptance criteria rather than from the code, and wire it into your pull request workflow as a required status check. One flow gated properly beats a broad suite that nobody trusts.

Install Kane CLI with npm install -g @testmuai/kane-cli and run the objective locally in headed mode first, so you can watch what the agent does before you make it a gate. The installation guide and CI recipes live in the TestMu AI documentation, and the verify loop for coding agents covers how to hand that same check to the agent so it self-corrects before opening a pull request.

Add the second flow only once the first has caught something. Continuous verification earns its place by blocking a real merge, and a team that has watched it do that once will extend it without being asked.