What Is an AI Audit?

What Is an AI Audit, and What Does It Examine?

An AI audit is a systematic, documented review in which someone independent of the build team compares evidence about an AI system, or about how an organization manages AI, with criteria fixed in advance, such as a legal duty, a clause of a standard or a policy line, and reports a finding per criterion.

The audit entry in NIST's glossary takes its wording from the glossary of the Committee on National Security Systems, CNSSI 4009-2022: a "systematic, independent and documented process for obtaining audit evidence and evaluating it objectively to determine the extent to which the audit criteria are fulfilled." The same entry quotes NIST SP 800-53 Rev. 5, where an audit is an "independent review and examination of records and activities."

• Criteria before evidence - the law, standard clause or policy line is written down before evidence is collected, so each finding can say how far one criterion was met.
• First, second or third party - per the same entry, an audit can be internal (first party) or external (second or third party); a certification body auditing against ISO/IEC 42001, for example, takes part as an outside third party.

The phrase has a second meaning, auditors using AI tools in their own audit work, which is a separate subject; this guide is about audits of AI systems.

How Is an AI Audit Different From a Pen Test, a Benchmark or a Certification?

A penetration test, a benchmark score or a certificate can each be used as audit evidence, and none of them is an audit on its own:

• Penetration test - NIST's glossary defines a penetration test as a test methodology in which assessors, "typically working under specific constraints, attempt to circumvent or defeat the security features of a system." Running one against an LLM application is covered in AI penetration testing.
• Benchmark - a fixed dataset used to score and compare models, and a benchmark score alone does not show whether one deployed system met your criteria.
• Certification - an ISO/IEC 42001 certificate records that a certification body audited an organization's AI management system against the standard. It is not a verdict on how any one model behaves.
• Audit - the review that weighs such evidence against the criteria and gives each criterion a result, repeated where a regime requires it: under Annex VII of the EU AI Act, a notified body "shall carry out periodic audits".

What Types of AI Audit Are There?

The five common types of AI audit are management-system, compliance, bias, technical and agent-behavior audits, each with its own criteria and evidence. One organization can need several: an ISO/IEC 42001 certificate covers its management system, while a hiring tool it uses in New York City still needs its own bias audit.

OWASP released the agentic Top 10 on December 9, 2025 and calls it "a globally peer-reviewed framework that identifies the most critical security risks" for agentic applications. Its entries can be written as criteria for the agent-behavior row.

Which Standards and Laws Shape an AI Audit in 2026?

Each status below was checked on official pages on October 28 and 29, 2026, using the IEC and BSI catalogs for the ISO/IEC standards. The middle column is a summary, not legal advice.

The Digital Omnibus on AI that amended the EU AI Act has been in force since July 27, 2026, according to the European Parliament's Legislative Observatory. For what Article 9 asks providers to test, see EU AI Act conformity testing.

Who Performs an AI Audit?

Internal auditors, certification bodies, EU notified bodies, independent auditors and government assessors all perform AI audits, and the criteria decide which of them you need. In NIST AI 100-1, MEASURE 1.3 calls for internal experts who were not front-line developers of the system, or independent assessors, to take part in regular assessments.

• Internal audit - for internal auditors, the Institute of Internal Auditors (IIA) publishes an AI audit framework, its updated Artificial Intelligence Auditing Framework, dated September 13, 2024.
• Certification bodies - they run ISO/IEC 42001 certification audits of an organization's AI management system, under the additional requirements that ISO/IEC 42006:2025 sets for such bodies.
• EU notified bodies - they assess high-risk systems whose route involves them and run the periodic audits Annex VII describes. For most Annex III systems (points 2 to 8), Article 43(2) has providers follow internal control, "which does not provide for the involvement of a notified body".
• Independent bias auditors - DCWP's FAQ on Local Law 144 calls a bias audit "an impartial evaluation by an independent auditor" and rules out anyone who works for the employer or the tool's vendor, was involved in using, developing or distributing the tool, or holds a direct or material indirect financial interest in either.
• Registered AI auditors - California's AB 1405 sets up a state registry of AI auditors, with rules on their independence, transparency and integrity.
• Government auditors and assessors - the U.S. Government Accountability Office published its accountability framework for AI (GAO-21-519SP) on June 30, 2021, organized around governance, data, performance and monitoring, with questions written for "entities, auditors, and third-party assessors."

How Do You Audit an AI System in 8 Steps?

This AI audit checklist shows how to audit AI one system at a time, and each step says what it means for the resume screener from the opening. The steps set out AI audit best practices in the order an audit applies them.

• Scope one system - record its name, version, owner and written purpose, since "our AI" cannot be audited; a checklist for AI auditing that the European Data Protection Board (EDPB) commissioned from outside experts and published in June 2024 asks for the same fields in its model card. For the screener: version 3.2, owned by talent acquisition, deciding which applications for New York City roles move to an interview.
• Fix the criteria - write each applicable law, standard clause and policy line as a statement that evidence can settle, dated before testing starts. For the screener, Local Law 144's minimum calculations are the criteria; the law sets no ratio threshold, so any review trigger comes from your own policy.
• Trace the data - record where training and input data came from, how it was processed and who can read it. For the screener, DCWP's FAQ requires the tool's historical data, allows test data only when history is insufficient, and bars imputed or inferred demographic data.
• Map tools and permissions - list every tool the system can call, what it can change and which actions need a person's approval. The screener only marks each application advance or hold, while an expense-approval agent can approve, return or route a claim, so each of its actions needs a limit and an approval rule. Actions without such limits are the risk OWASP files under LLM06:2025 Excessive Agency.
• Test before release - run functional, bias and adversarial tests against a staging copy and keep the results with the build. NIST AI 100-1 states that "AI systems should be tested before their deployment and regularly while in operation", and tests for tool misuse and injected instructions can follow the test plan in AI agent red teaming. For the screener, the bias part of this step is the impact-ratio calculation below, which runs on the tool's historical data rather than on a staging copy.
• Keep the evidence - store test runs, tool-call logs, changed records, approvals and failed runs under the version they tested, which for the screener means the data extract, the calculation and its output. Capturing tool calls with their inputs and results is covered in AI agent tracing.
• Report per criterion - give each criterion one result, met, not met or could not be verified, with the evidence cited. MEASURE 1.1 of the NIST AI risk management framework asks for documentation of each risk that cannot be measured or will not be, and the third result is that record. The screener's results feed the summary that Local Law 144 requires to be published.
• Set re-audit triggers - decide which changes, such as a new model, prompt, tool or policy, reopen which criteria. Under Article 43(4) of the EU AI Act, a high-risk system that undergoes a "substantial modification" goes through conformity assessment again, and DCWP lets an employer rely on a bias audit for one year. The guide to AI agent validation has a table of the changes that should trigger revalidation of an agent.

LL144 bias audit: illustrative numbers, not real data
Resume screener; selected = moved to interview

Sex categories
Category                   Applied Selected   Rate  Impact
Male                           590      236  40.0%   1.000
Female                         560      196  35.0%   0.875

Race/ethnicity categories
Category                   Applied Selected   Rate  Impact
Hispanic or Latino             260       91  35.0%   0.831
White                          356      150  42.1%   1.000
Black or African American      220       70  31.8%   0.755
Asian                          190       76  40.0%   0.949
Native Hawaiian/Pac. Isl.       14        5  35.7%   excl.
Native Am./Alaska Native        30       10  33.3%   0.791
Two or More Races               80       30  37.5%   0.890

Applicants in the calculations: 1150
Unknown category, not in the rates: 50
excl. = 1.2% of the data, under 2%: no ratio
Rows after Hispanic or Latino: Not Hispanic or Latino

How Is an AI Agent Audit Different From Other AI Audits?

Take the expense-approval agent from step 4: it checks claims against the expense policy, approves those within its limit and routes the rest to a manager. The EDPB-published checklist asks whether an AI component interacts with other systems or applications, its own or a third party's, and for an agent, whose decisions land as records in another system, that question is where an AI agent audit starts:

• Claim records - a claim is approved when the expense system's status and history say so, whatever the agent's summary reports, so the audit reads the claim records back.
• Later steps - an agent that approves an over-limit claim and then routes the same claim to a manager can end with a message that reads correctly, while the approval stays in the claim's history.
• Permission limits - an approval permission capped at the agent's limit still lets it approve a duplicate claim or one missing a required receipt, so the audit also tests the decisions made inside that limit.
• Injected instructions - a description field telling the agent that the claim is pre-approved is an injected instruction, and the adversarial tests from step 5 should include it.

Agents can also report approvals that never happened; such false completion reports are covered in AI agent hallucination. How completely a reviewer can rebuild one run from records the agent cannot alter is a question of AI auditability, which the guide to agentic AI governance covers together with the records an agent's audit trail should hold.

Recorded tool calls give a reviewer something to check a claimed action against, and designing agents that leave such records is the subject of AI verifiability.

How Do AI Audit, AI Assurance and AI Governance Differ?

AI governance sets the rules, and an AI audit is one of the AI assurance techniques that check a system against them:

• AI governance - the rules, owners and risk tiers an organization applies to AI; turning the laws and standards above into that structure is the job of an AI governance framework.
• AI assurance - the wider family of techniques for showing that a system meets its criteria; a UK government portfolio of AI assurance techniques, covered in the guide to AI assurance, lists bias audits and compliance audits among them.
• AI audit - one of those techniques, in which someone independent of the build team checks evidence against criteria fixed before the review, within an agreed scope, and reports to whoever commissioned it.

NIST AI 100-1 gives the reason for keeping the review apart from the build team: "Processes for independent review can improve the effectiveness of testing and can mitigate internal biases and potential conflicts of interest."

What Evidence Will an AI Auditor Ask For?

An AI auditor asks for dated evidence, per system and per criterion:

• Criteria and their sources - each criterion traced to a law, a standard clause or a policy line, with the date it was fixed.
• System identity at test time - name, version and owner, and for an agent also the model, prompt and tool list that each test ran against.
• Read-back records - for agent tests, the tool calls and the state read back from the systems the agent changed, such as each claim's status history.
• Runs and results - every test run with its date, version and environment, failed runs included, and a result per criterion, with the evidence that would settle each criterion that could not be verified.
• Approvals, exceptions and changes - who approved a release or an exception, and when, plus the model, prompt, tool or policy changes since the last audit and the criteria each one reopened.
• Published bias summary - for a New York City hiring tool, DCWP's FAQ lists what the public summary holds: the date of the most recent bias audit, the source and explanation of the data, the number of individuals in an unknown category, and the number of applicants, selection or scoring rates and impact ratios for all categories.

Producing Pre-Release Agent Evidence for an AI Audit With Agent Assurance

For the agent-behavior part of an AI audit, TestMu AI's Agent Assurance puts agents that take actions, like the expense-approval agent, through tests before release. What it produces is evidence for an auditor to review; Agent Assurance is not an auditor itself.

Through its terminal client, Rook CLI, it takes the agent's code or spec, builds functional and adversarial scenarios by default, each with acceptance criteria, and calls the real agent through a profile: a command, an HTTP endpoint or an MCP tool, driven by hook scripts you can review. Each criterion is graded from evidence of what the run did, never from the agent's own account of it:

• Output and actions - the agent's replies are graded from quotes of its actual output, and its actions from the tool calls and state changes the profile's hooks report, or from a read-only lookup through a tool you approve. Model judges decide some criteria, and the agent's own report of an action is never taken as proof that it happened.
• Unverified criteria - each criterion gets one verdict out of Pass, Fail and Unable to Verify, and the third is reported apart from the other two, so an auditor can see which criteria nobody could check. How much a run can see depends on what the hooks report and which read-only tools you approve.
• Attack scenarios - adversarial scenarios are mapped to entries in MITRE ATLAS, the OWASP Top 10 for Agentic Applications (2026) and the OWASP Top 10 for LLM Applications (2025), and the report's OWASP coverage table shows security coverage rather than compliance; a zero row there means only that no scenario covered the entry.
• Traceable run files - every run is stored as plain files in your project, so an auditor can follow each verdict to its quoted evidence and to that run's own copy of the scenarios, agent version and profile revision. The run plan is written to disk as a record of what an unattended run was set to do, and editing a scenario later leaves an earlier run's copy unchanged.

Most eval and observability tools score what your agent said and recorded. Agent Assurance checks what the run changed, and it names each criterion it could not verify.

The agent's writes during a test are real and are not rolled back, so run the tests against a staging copy of the expense system with synthetic claims, and redact personal data and credentials from the run files before an auditor sees them. Agent Assurance does not observe production traffic or stop an agent's action at runtime.

Whether your organization or its agent meets ISO/IEC 42001, the EU AI Act, the NIST AI RMF or Local Law 144 is for your compliance team and auditors to decide: Agent Assurance tests and reports, maps to none of those texts and certifies nothing.

Agent Assurance is generally available on self-serve plans, and a team can start with the 14-day free trial, which includes 4,000 credits. Until the TestMu AI account holds a current Agent Assurance plan or trial, Rook CLI makes no model call and spends no credits; the pricing page lists the Agent Assurance plans.

Install Rook CLI with npm, then add its skill to Claude Code if your team builds agents there:

npm install -g @testmuai/rook@latest
npx @testmuai/rook-skill@latest install \
  --agent claude-code

• Node.js 20 for npm - install Node.js 20 or newer first. Rook CLI installs on macOS and Linux, and 64-bit Windows through npm or WSL.
• Homebrew and shell installer - neither needs Node, and both cover macOS and Linux.
• Node.js 22 for the skill - Node.js 22 or newer is required by the skill installer, npx @testmuai/rook-skill.

The docs on Agent Assurance results and evidence show the files a run writes and what each one records. For an Enterprise plan, Book a Demo.

Which AI System Should You Audit First?

Start your first AI audit with the system that has the nearest legal deadline: write its scope (step 1) and its criteria (step 2), and date both before anyone collects evidence. Run the step 5 tests on a staging copy, then send the auditor the evidence listed above before the first meeting, with each criterion still marked could not be verified next to the record that would settle it.

If the system is an agent that edits records, such as the expense-approval agent, follow the Rook CLI installation guide, then give Agent Assurance the policy your criteria quote, so its scenarios test the agent against the same lines the auditor will read.

Note: This article was researched and drafted with AI assistance, then reviewed, fact-checked, and published by Maneesh Sharma, Chief Operating Officer at TestMu AI and the article's author of record, whose listed expertise includes software testing and enterprise software. Sirajuddin Khan, Vice President of Product Management at TestMu AI, reviewed it. Every statistic, link, and product claim was verified against primary sources: the IEC and BSI catalogs for the three ISO/IEC standards, NIST, the European Union's official pages, New York City's Department of Consumer and Worker Protection, the Governor of California, OWASP, the U.S. Government Accountability Office, the IIA, ISACA, the U.S. Bureau of Labor Statistics, the EDPB and TestMu AI's product documentation. Read our editorial process and AI use policy for details.