AI Incident Reporting Starts Before the Incident

What Is AI Incident Reporting?

AI incident reporting is the obligation (whether legal or contractual) to report to regulators, customers or upstream providers that an AI system has caused or contributed to harm, within a defined time period after you become aware of it. It is the final step in the AI incident response process and is the step that creates the public record.

The definitions are important because they determine which events trigger the clock. Under the EU AI Act, a 'serious incident' is defined as an incident or malfunction of an AI system which directly or indirectly results in death or serious harm to health, serious and irreversible disruption of critical infrastructure, infringement of fundamental rights obligations under Union law, or serious harm to property or the environment (Article 3(49)).

NIST's Generative AI profile lists incident disclosure as one of the four primary considerations for generative AI risk, and the recommended actions are to report GAI incidents "in compliance with legal and regulatory requirements".

Note the word 'indirectly'. An agent needn't be the final actor in the chain for its behavior to count.

How Quickly Do You Have to Report an AI Incident?

Faster than most enterprises could ever hope to rebuild. The windows below start at awareness, and none of them stop while your team is working out what the agent did.

Sources: EU AI Act Art 73, Art 26, Cal. Bus. & Prof. Code §22757.13, Morrison Foerster on the RAISE Act amendment, 2026.

Two very honest caveats a reasonably skeptical general counsel would raise.

Firstly, timing. The EU Digital Omnibus (Regulation (EU) 2026/1744) has moved the high-risk obligations back: standalone high-risk systems will be subject to them from 2 December 2027 and high-risk systems embedded in a product from 2 August 2028.

Article 73 has no separate start date, so in reality its clock will start for those systems when they become high-risk under the regulation (exact date to be confirmed with legal advice). It's a delay to allow time to build. It doesn't extend the windows once they do apply.

Secondly, scope. SB 53 and the RAISE Act are about the frontier model developers, not the retailer, the airline or the manufacturer who are running the agents on top of those models. The obligations do, however, flow down through the contracts.

A frontier developer with a 72-hour window is going to need its enterprise customers to provide the evidence in that same 72-hour window. Article 73(5) allows a reporter to file an initial report which is incomplete and then complete it later. That helps a bit, but only if you actually have something of value to report in the first instance.

Why Can't Most Companies Report an AI Agent Incident in a Timely Manner?

As agents act through their legitimate permissions, the only real record of why they acted is often tucked away in the agent's own context and the vendor's logs. Reconstruction rather than detection is the bottleneck.

The readiness data is very thin and not at all encouraging. ISACA's September 2026 survey of over 1,800 cybersecurity professionals revealed that 64% of enterprises had run no AI-related incident response exercises at all and only 8% ran AI-specific exercises on a regular basis.

IBM's 2025 Cost of a Data Breach study found that 13% of organizations had experienced a breach of an AI model or application and of those, 97% reported that they had no AI access controls in place.

Public disclosure is just as patchy. Eon's November 2026 review of 33 public cases of AI agents breaking production systems found that 29 of those were self-reported, and the public record is silent as to how 16 of them concluded. That's what AI incident disclosure looks like without any evidence at all: just anecdotes, many of them with no ending.

Our colleagues covered the operational side last month - AI agent incidents don't look like outages. The system is up. The agent is doing something it was allowed to do, just wrong.

That's precisely why a reporting regime based on "what happened and what was impacted" is very difficult to satisfy with an uptime dashboard.

Is AI Incident Response Becoming a Security Product?

Detection is. Evidence isn't. On 3 November 2026, CrowdStrike announced a definitive agreement to acquire Fiddler (an AI agent observability and explainability company) to bring agent observability to its Falcon platform.

We read that as a very clear signal that agent observability is being brought into the security stack, alongside endpoint and identity.

That's good for detection. Security teams are already doing incident response, and providing them with agent telemetry closes a real gap.

But detection and disclosure are answering two very different questions. Detection is asking, "is something wrong at this very moment?"

Disclosure is asking "what exactly did the system do, on what authority, to whom, and how do you know?" The second question requires evidence that:

• was designed before release, not assembled during the incident;
• does not depend on the model under review to explain itself;
• sits in the deploying company's control plane where their lawyers and auditors can get to it.

That's where we think AI governance budgets will go over the next two years actually (not more dashboards, but reportable evidence).

The 48-Hour Disclosure Test: A Framework for AI Incident Reporting Readiness

The 48-hour disclosure test asks whether, in the shortest time frame allowed by the EU AI Act (i.e., 48 hours), your organization could have answered six questions about the agent's action, using only evidence that existed prior to the question being asked. Each of the six questions corresponds to one of the AI trust principles that we adhere to at TestMu AI.

Score one point for each question you can answer with evidence that already existed. Six is disclosure-ready.

Four or five means you can file an initial report and complete it. Three or below, and your first report will be a prediction.

What Does an AI Incident Reporting Maturity Model Look Like?

We see five levels, and the key jump is from Level 1 to Level 2 (from logs you keep to evidence you can report).

Level 4 is where AI incident management ceases to be a cost center. Every incident is now a test that runs prior to the next release, so the same failure won't be able to ship again without being caught.

Where Should AI Governance Budgets Go to Fix AI Incident Reporting?

Invest in making incidents reconstructable before you invest in detecting them any faster. Detection is just about consolidating into the platforms you're already paying for. Reconstruction is not.

Our view, for a CIO setting 2027 priorities:

• Put an agent release gate in front of production. Nothing gets shipped by an agent without a recorded run against its failure and attack use cases. That's Level 3 in practice, and it turns Q5 from a guess into a lookup.
• Separate the verifier from the model. Budget for verification that runs where the model runs but is independent of it. That's the AI assurance layer, separate from agent observability.
• Write your reporting runbook against the shortest clock. Name the owner, the template and the evidence sources for a two-day report and then drill it. The European Commission has already consulted on a reporting template for serious incidents, so take that as your starting point.
• Push the evidence requirements into the vendor contracts. For every agent vendor, ask exactly how you get a readable action trace in your systems within 24 hours of a request.

Prediction: by the time stand-alone high-risk obligations come into play in December 2027, we're going to expect the enterprise AI procurement questionnaires to be asking for a sample incident evidence pack (the way security questionnaires currently ask for a pen test summary).

The trend lines behind that are the reporting clocks which are already enshrined in law, the readiness gaps being measured by ISACA and IBM, and the security vendors going into agent observability via purchase.

How TestMu AI Approaches Pre-Production Evidence

TestMu AI builds the independent verification layer so that incident reporting has something genuine to start from. Three elements of the platform map to the 48-hour disclosure test:

• Agent Assurance, delivered by Rook, runs your agents against a test suite before release, in CI. It grades what an agent actually changed rather than what it says it did, checks its tool calls against the agent's declared tool surface, and keeps per-criterion evidence as plain files, including a record of what could not be verified.
• Agent Testing covers the human-to-agent channels (chat, voice, phone and IVR, video and image agents) where the human agent interacts with the real customer.
• HyperExecute will orchestrate those suites at scale so that a new incident scenario can become a regression test that runs on every release. The same cycle can be driven by developers and coding agents from the terminal via Kane CLI.

Our testing is deterministic in that the same scenario will always produce the same verdict, which is why it's usable as evidence. We apply the same test to ourselves as we do to TestMu AI.

TestMu AI's verification doesn't rely on the model that we're looking at, and the evidence it generates is the evidence that we want to hand to the customer, not the evidence that we keep.

Conclusion: AI Incident Reporting Is Decided at Release Time

AI incident reporting is all about the legal deadline. In reality, it's a test of the engineering decisions that were made months prior to the incident. The clock is always ticking, the definitions are very broad, and the agent that caused the incident is the absolute worst possible witness.

Run the 48-hour disclosure test on your three most important agents this quarter. If you score 3 or below, you don't need a better incident response plan first. You need to have the evidence in place before anyone even asks for it.

Shipping agents that other agents use? Rook by TestMu AI does Agent Assurance: agents testing agents before release, with no human in the loop.

To run your first test, install Agent Assurance with npm install -g @testmuai/rook@latest command, or

book a demo

for a full walkthrough of the product.