AI Red Teaming Is a Regression Suite, Not a Report: The TRACE Loop for AI Agents

What Is AI Red Teaming and What Does That Mean for AI Agents?

AI red teaming is effectively adversarial testing of an AI system where humans and tools are used as attackers to try and get the system to breach its own policies. The breaches are then remediated before a real attacker can find them.

The UK government's initial guidance on frontier AI referred to it as an exercise in looking at the system from the perspective of an adversary and seeing how it could be attacked or abused (GOV.UK, 2023).

LLM red teaming used to be all about chat jailbreaks. Agents have broadened the scope of the attack considerably. An agent will go out and read web pages, emails, tickets, etc.

It will call tools, maintain memory, and pass work on to other agents. Each of those is an input that an attacker can write to. That's why indirect prompt injection (where the malicious instruction is hidden within the content the agent retrieves) has become such a major concern in agentic AI security.

The scale of the problem is now quantified. In a public competition sponsored by the UK AI Security Institute in conjunction with a number of leading labs on Gray Swan's arena, 1.8 million prompt injection attacks were submitted against 22 frontier AI agents across 44 different deployment scenarios. Over 60k of the attacks were successful in causing policy violations, and almost all of the agents were found to have violated policy for the majority of the behaviours in the 10 to 100 query range (Zou et al., arXiv, 2025).

Defences are improving. Anthropic reported in Nov 2025 that their improved defences had reduced the browser use attack success rate of Claude Opus 4.5 to around 1% against an adaptive attacker (Anthropic, 2025). A lab benchmark is still not your reality.

Why Does a One-Off AI Red Teaming Report Go Stale?

The system you tested is in a constant state of flux. A model update, a prompt tweak, a new tool, a new MCP server, or a new retrieval index could all introduce new failure modes. Your threat model will still be based on the old report, just won't be able to cover the new system.

• Newer is not safer by default. A longitudinal study of successive GPT versions found that model updates didn't necessarily lead to any improvement in adversarial robustness (Liu et al., arXiv). The Gray Swan competition paper likewise found very little correlation between agent robustness and model size, capability or inference time compute.
• The attack catalogue is still growing. On 29th October, CrowdStrike took their public prompt injection taxonomy to 222 techniques, adding in attacks that are specifically targeting the way the agents plan and deconstruct goals (CrowdStrike, 2026).
• The labs say it won't be finished. OpenAI in Dec 2025 stated that prompt injection is "unlikely to ever be fully 'solved'" (OpenAI, 2025). Microsoft's AI Red Team, having tested over 100 generative AI products, has it as one of their key learnings that it's impossible to ever fully secure an AI system (Microsoft, arXiv, 2025).
• Your own change rate is the multiplier. If your team is editing prompts or adding tools between red teaming engagements, then each of those changes will go out untested against the attacks you already know about.

Red teaming can't just be an event. You need a test suite with a constantly growing set of test cases.

Red-Team Report vs AI Red Teaming Tools vs Regression-Gated Red Teaming

The three approaches deal with different questions.

Open-source tools such as PyRIT, garak and Promptfoo can already run in CI. The difference TRACE adds is not automation. It is pinned, versioned fixtures with action-level assertions and evidence you own.

The gate's job is to re-check every known attack, and pinned variants of it, on every change.

The TRACE Loop: How Do You Run AI Red Teaming as a Regression Suite?

The TRACE loop is a 5-step method which turns AI red teaming from a one-off report into a release gate: Threat model authority, Run independent attacks, Archive evidence, Convert findings to fixtures, Enforce at the gate. Each step maps to one of the AI trust principles we follow at TestMu AI.

C: Convert Every Finding Into a Deterministic Fixture

Each successful attack that survives the T step becomes a regression test with three fixed parts:

• A frozen payload. The exact attack plus a number of variants (rephrasings, encodings, different carriers such as a PDF or a calendar notification). The variants are pinned per release and then refreshed on a schedule so the suite doesn't overfit to the known strings.
• A pinned environment. Seeded data, recorded or mocked tool responses, and a fixed retrieval corpus so the only thing that varies between runs is the system under test. Re-record your mocks when you're testing a new model that calls a different set of tools, and maintain a smaller number of live staging runs so the mocks don't completely hide the real behaviour.
• Binary assertions on actions. Did the agent invoke the refund tool? Did any customer field go out of bounds? Did it write to a path outside its scope? Assert on what the agent actually did, not on how it described it.

The test design is deterministic even if the model's sampling isn't: same payload, same environment, same pass rule. If one run is going to miss a rare failure, then you need to run each critical fixture enough times to bound the rate you can tolerate.

Zero failures in 300 runs will bound the violation rate at less than 1% (the "rule of three") at 95% confidence.

Record the run count with the evidence. Where content requires judgement, then use an independent judge and record their rationale.

A passing fixture shows that the known attack still fails. It doesn't prove the weakness behind it is gone. That's why the loop has adaptive red teaming (R) running alongside the gate.

E: Enforce at the Release Gate, Including the Off Switch

The fixture suite runs on every change you have control over: model version, system prompt, tool schema, MCP server, retrieval index or guardrail config. A critical failure will block the release.

Pin model versions rather than use floating aliases and run the suite on a schedule to catch any drift you don't have control over (e.g. a vendor-side model update or the continuously refreshed index). This is continuous system testing.

Add one fixture class that is very easy to overlook: containment. You need to test that an authorised person can pause or stop the agent in the middle of a task and that the agent cannot switch off its own oversight.

OpenAI's misalignment report (updated October 20) describes two cases of internal agents where they had disabled the automatic alerts to the security team (one case) and had removed a monitoring integration (the other case).

These were misalignment incidents, not attacks, so you do need to test the agent's own behaviour in the context of a range of different tasks. If the kill switch isn't under test, then it's just a hope, not a control.

Who Should Red-Team an AI Agent? The Model Itself? The Vendor? Or an Independent Party?

Not the model under test, and not merely its vendor's built-in monitor. Built-in guardrails and monitors are of course valuable runtime controls, but they are models as well.

The Anthropic Fellows study is the clearest example of this. In their September 2026 snapshot (where the attacker typically knew how the monitor worked), Claude Code's Auto Mode monitor and a version of Codex's Guardian both failed to block over 55% of the attacks. A better monitor reduced attack success by over 50pp while blocking less than 1% of the benign actions (Anthropic Alignment, 2026).

That's real progress and a case for treating AI guardrails as a target in your suite rather than a substitute for it.

This is our independent audit principle: the audit must not be reliant on the model under audit. Two edge cases to be very clear about:

• In-house builds. You don't need to buy a tool in order to be independent. You need an attacker and a judge who don't share the target's blind spots and a gate that's owned (or signed off) by someone outside of the delivery team.
• Vendors (including us) ask you three questions: which model family are the attacker and judge running? Can you pick a different one? Can every verdict, rationale and hash be exported to your own store and then checked on your own?

AI Red Teaming Maturity Model: Where Is Your Team?

Levels describe capability, not history; a team can skip L1.

L4 includes incident disclosure: when an attack gets through in production, tell affected parties quickly, add the attack as a fixture, and share the lesson so others can test for it.

What Do the Regulators and Standards Expect From AI Red Teaming?

Documented adversarial testing for a number of model providers, attack resilience for high-risk systems and a regular testing cadence (as voluntary guidance). None of them mandate a per-release regression gate, but taken together they are a good reason to have one.

• EU AI Act, Article 55(1)(a). Providers of general purpose AI models which present a systemic risk must assess their models "including conducting and documenting adversarial testing" (EU AI Act Service Desk). This applies to a very small number of model providers indeed, not the vast majority of enterprises. It does, though, set the template: conducting and documenting.
• EU AI Act, Article 15(5): High-risk AI systems must be resistant to any attempt to alter their use, their outputs or their performance in an unauthorized way. Where appropriate, technical measures should be in place to address data poisoning, model poisoning, adversarial examples, confidentiality attacks and model flaws (EU AI Act Service Desk). Following the Digital Omnibus, the Commission states that the high-risk rules will apply from 2 December 2027 and from 2 August 2028 for AI systems embedded in regulated products (European Commission).
• NIST AI 600-1 (Generative AI Profile) recommends AI red-teaming against attacks such as prompt injection and adversarial testing "at a regular cadence" (NIST, 2024).

From the WEF's Global Cybersecurity Outlook 2026, the proportion of respondents who said they had a process to assess the security of AI tools before deploying them has gone up from 37% in 2025 to 64% in 2026. That said, around a third still have no process at all to verify the security of AI before it is actually deployed (WEF, 2026).

Access is the other issue. IBM's 2025 Cost of a Data Breach research found that 13% of organisations had experienced a breach of an AI model or AI application and 97% of those had no AI access controls (IBM, 2025). That is precisely the problem that the authority map in step T makes clear.

Gartner predicts that by 2028, 25% of enterprise GenAI applications will have at least five minor security incidents per year (up from 9% in 2025) (Gartner, 2026).

How TestMu AI Runs the TRACE Loop

At TestMu AI, we build an independent verification layer in the form of tests that produce evidence rather than opinions. The TRACE loop maps to the three kinds of software interaction that we test.

• Human to agent (chat, voice, IVR, WhatsApp). Agent Testing is where you run scripted conversations against the agents that are going to be facing the customer. Adversarial personas and injection attempts would belong in that suite as fixtures.
• Agent to agent. Agent Assurance delivered by Rook has agents testing agents prior to release with no human handshake in the test run. Your people own the findings and the gate thresholds.
• Agents behind a web or mobile interface. SDETs can author and run their regression tests from the terminal via Kane CLI, and manual testers can write them in plain language via KaneAI.
• Running the gate. HyperExecute orchestrates the fixture suite in parallel to shorten the gate, subject to your model provider's rate limits.

Our testing is deterministic (fixed inputs, pinned environments, pass/fail assertions on actions). We apply the same to ourselves in that the agent under test will never grade its own run and the evidence is designed to be in the customer's own control plane. Ask us the three vendor questions above as you would any other vendor.

Setup details are in our docs.

Three Predictions for AI Red Teaming

These are our views, each tied to a trend cited above.

• By the end of 2027, we expect regulated buyers to be asking for red team regression evidence per release rather than just a periodic report. Per-release evidence is the most straightforward way of demonstrating the documented testing and attack resilience that the EU AI Act requires of them prior to the 2nd December 2027 high-risk date.
• The fixture and evidence layer becomes where buyers compare vendors. CrowdStrike's agreements to acquire both Dreadnode (attack generation) and Fiddler (observability) suggest security platforms want both ends. One deal is not a trend, but it points to the gap between them.
• Monitors and guardrails become their own test target. The Anthropic study shows monitors can be bypassed, and OpenAI's report shows agents can work around oversight with no attacker involved. Suites will test the agent and its overseer separately.

Conclusion: AI Red Teaming That Compounds

AI red teaming earns its keep through compounding. A regression suite will re-run the known findings across every model swap, every prompt edit, and every new tool, and leave a trail of evidence that a CIO, auditor or regulator can re-play.

Start small this week. Take your last 10 red team findings. Close the ones you can by reducing the agent's authority.

Freeze the rest as fixtures with action-level assertions (assessed independently and archived) and make them block your next release. That is L3 at the heart of the maturity ladder and the basis for everything else in AI assurance.

Related readings:

• What is an AI Audit?
• What is an Agent Harness?
• What is AI Verifiability?
• What is AI Trust?

Shipping agents that other agents use? Rook by TestMu AI does Agent Assurance: agents testing agents before release, with no human in the loop.

To run your first test, install Agent Assurance with npm install -g @testmuai/rook@latest command, or

book a demo

for a full walkthrough of the product.