How to Build an Agent-Native CI Pipeline

What Actually Changes

Not the speed of the eventual fix. Which step a human is doing.

In a conventional pipeline, a human opens a red check, reads a log, forms a hypothesis, and only then starts fixing. Agent-native CI moves the first three steps, reading, classifying, and hypothesizing, onto the agent. The human's first look at the problem is already a diagnosis with evidence attached, not a wall of log text.

• Reading - the agent pulls the failing run's artifacts instead of a human scrolling a log.
• Classifying - the agent sorts the failure into a category before a human has to guess.
• Proposing - the agent drafts a fix or a root cause, delivered as something to review, not something to invent from scratch.

Notification vs Diagnosis

A Slack message that says a build failed is a notification. It tells you something happened. It does not tell you why.

The gap between those two is where an agent earns its place in the pipeline, and it depends entirely on what evidence the agent can actually reach. A plain-text log forces the same reconstruction work a human would do, just performed by a model instead. Structured artifacts, screenshots, a HAR network log, console output tied to the exact failing step, let the agent skip reconstruction and go straight to classification.

Kane CLI from TestMu AI builds this pattern into every run rather than leaving it to whoever wires the pipeline. Each run seals an evidence pack, per-step screenshots, a HAR network log, console NDJSON, and a structured failure record, into one file under .testmuai/evidence/. That pack is what an agent-native triage step should be reading: not a log to parse, but a record already shaped for classification.

The Failure Classes a Triage Step Needs

A useful classification separates what needs a code fix from what needs nothing but a rerun.

TestMu AI's HyperExecute already runs a version of this classification in production. It collects Selenium logs, network logs, console logs, screenshots, and video for every test, and its AI root cause analysis reads that evidence to name a cause, the same job a custom agent-native triage step would otherwise have to build from scratch.

Wiring the Loop With workflow_run

GitHub's events-that-trigger-workflows documentation defines workflow_run as firing "when a workflow run is requested or completed," which is what lets a triage workflow act on a test workflow's outcome instead of running blind alongside it.

on:
  workflow_run:
    workflows: ["Test Suite"]
    types: [completed]

jobs:
  triage:
    if: ${{ github.event.workflow_run.conclusion == 'failure' }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v5
        with:
          run-id: ${{ github.event.workflow_run.id }}
          github-token: ${{ secrets.GITHUB_TOKEN }}

Two details in that documentation change how you should write this workflow. First, the conclusion field and the run ID are both on the event payload, so the triage job can gate on failure and fetch exactly that run's artifacts without a separate API call. Second, and easy to miss: the triggered workflow can access secrets and a write token even if the workflow that triggered it could not. That second fact is the whole reason the guardrails section below exists.

What Stays Human: The Merge Gate

Every part of this loop that ends in a pull request is optional to speed up. The merge is not optional to keep human.

• The agent proposes - a classification, a root cause, or a draft fix, delivered as a PR.
• A human reviews - the diagnosis and the diff, the same review any teammate's PR would get.
• Only a human merges - no configuration in this pattern grants the agent merge rights, by design rather than by omission.

This is the same line our guide to quality gates for AI pull requests draws from the other direction: an agent's own PR needs the same gate as a human's, and an agent's triage output needs the same review as a human's diagnosis would.

Guardrails Against a Hidden Regression

The failure mode worth designing around is not the agent missing a bug. It is the agent making a red check turn green without the underlying bug actually being fixed.

• Path allowlist - the triage job's write access excludes the test files it is diagnosing, so it cannot loosen the assertion that failed.
• Proof by rerun - a proposed fix is only credible once the original failing check is rerun and genuinely passes, not once the diff looks plausible.
• Scoped tokens - since workflow_run grants secrets and write access the triggering workflow may not have had, the triage job's token should be scoped to exactly what triage needs, not inherited wholesale.
• PR-only delivery - every output lands as a reviewable pull request, never a direct commit to a protected branch.

The scoped-token guardrail is the one teams skip most often, because it requires reading the workflow_run permissions model closely enough to notice the elevation happens at all.

A Worked Example

A browser check fails on a pull request. The test workflow completes with conclusion failure, and the triage workflow above fires.

1. Test Suite workflow runs, a checkout-flow assertion fails, conclusion: failure
2. workflow_run fires the triage job, downloads the run's artifacts
3. Triage reads the Kane CLI evidence pack: a screenshot showing a 500 response
   on POST /checkout, the HAR log confirming the same request server-side
4. Classification: application bug, not a flaky selector or infra failure
5. Triage opens a PR: "checkout POST fails with 500 after promo-code refactor,
   see evidence pack for request/response detail"
6. A human reviews the PR, confirms the diagnosis, approves the fix
7. Merge re-triggers Test Suite; the original failing check now passes

Step 3 is where the whole argument for artifacts-over-logs pays off. A screenshot and a matching HAR entry name the failure in one look; a plain-text log of the same failure would need the agent, or the human, to reconstruct the same conclusion from timestamps and stack traces.

What This Does Not Fix

Agent-native triage classifies and explains failures that a check actually caught. It has no opinion on flows nobody thought to test.

• Coverage gaps - a flow with no assertion produces no failure, so there is nothing for the loop to read.
• Ambiguous requirements - a triage step can say what broke, not decide what the feature was supposed to do in the first place.
• The three-level ceiling - explore, classify, and fix already uses most of workflow_run's chaining budget, so a design that assumes a fourth automated hop needs a different trigger entirely.

None of that is a flaw in the pattern. It is the boundary of what CI-stage triage was ever going to solve, and it is why the coverage question comes before the triage question. Our breakdown of agent-run test cost vs plain CI covers the budget side of deciding how much of a suite deserves this treatment in the first place. Start with the Kane CLI documentation to give your own triage step evidence worth reading.