What Is AI Security? Risks by Layer, Frameworks and Testing

What Is AI Security, and What Does It Protect?

AI security is the practice of protecting AI systems, their training and retrieval data, models, applications, agents and the infrastructure that runs them, from attacks that poison, steal, misuse or manipulate them. It keeps those parts confidential, intact and available, and it covers ordinary software flaws in AI tooling as well as attacks that only work against machine learning.

NIST draws the line between the two kinds of risk in Appendix B of its AI Risk Management Framework. It says guidance written before the framework cannot "comprehensively address security concerns related to evasion, model extraction, membership inference, availability, or other machine learning attacks" or "account for the complex attack surface of AI systems".

• Assets - training and fine-tuning data, retrieval indexes, model weights, system prompts, tool credentials, and the records an agent can change.
• Attackers - anyone who can write to a data source the system reads, query its API, publish a package or model it loads, or talk to it directly.
• Outcomes to prevent - leaked data or weights, a model that behaves differently on attacker-chosen inputs, and actions taken on an attacker's behalf.

AI security and AI safety are separate disciplines that share some tools. The NIST AI RMF 1.0 (NIST AI 100-1) treats security as one property of trustworthy AI among several, with safety, privacy and fairness next to it, and adds that "Creating trustworthy AI requires balancing each of these characteristics based on the AI system's context of use." Security work asks who could attack the system; safety work asks what harm the system could cause on its own.

What Is the Difference Between Securing AI and Using AI for Security?

Securing AI protects the AI systems you build or run. AI for security uses AI to do security work on any system, such as detecting threats, triaging alerts or reviewing code. Search results for "what is AI security" mix the two, so check which meaning a source uses before you compare its advice.

NIST keeps them apart in the preliminary draft of its Cyber AI Profile (NIST IR 8596), published in December 2025 and still a preliminary draft when read on 2026-10-19. It organizes AI cybersecurity into three focus areas:

The Thwart area matters because attackers use AI too. In LLM Agents can Autonomously Exploit One-day Vulnerabilities (Fang and colleagues, April 2024 preprint), a GPT-4 agent given the CVE description exploited 87% of 15 one-day vulnerabilities, against 7% without the description and 0% for GPT-3.5, the open-source models tested, ZAP and Metasploit.

For the AI-for-security meaning, the guide to security testing has a section on how AI is changing security testing work.

Why Do AI Systems Need Their Own Security Practice?

AI systems keep every classic weakness of the software around them and add properties that classic controls were not built for. Greshake and colleagues put the core one plainly in Not what you've signed up for (2023): "LLM-Integrated Applications blur the line between data and instructions."

• Data is also code - a model treats any text that reaches its context as a possible instruction, so a document, email or tool result can steer it. The same paper showed indirect prompt injection working against Bing's GPT-4 powered Chat with no direct interface to the target.
• Learned behavior - the logic sits in weights and data, so a poisoned dataset or a swapped model file changes behavior without a code change anyone can review.
• Outputs vary between runs - the same attack can fail on most runs and succeed on one, so a single clean test run proves little and results are rates.
• Agents act - once a model can call tools, a manipulated answer becomes a refund, a deleted file or an email to an outsider, and the damage happens in other systems.

The classic weaknesses do not go away. A model's output that reaches a shell, a database query or a browser carries the old injection risks with it, which is why AI security programs keep their application security testing and add AI-specific tests on top.

What Are the Main AI Security Risks at Each Layer of an AI System?

An AI system has five layers worth separating, because each has its own attackers, standards and tests. The table maps each layer to an entry in the OWASP lists and a technique in MITRE ATLAS, MITRE's knowledge base of attacks on AI systems; technique names are from the ATLAS 2026.09 data release, read on 2026-10-19.

Where Do Security Flaws in Open-Source AI Packages Come From?

In 13 open-source AI packages, every advisory filed since January 2025 that carries a weakness ID names a classic application security weakness, and none names an AI-specific one. That is the result of a count of public advisories run for this guide on 2026-10-19 through the OSV.dev API, queried without an API key; the figures below are the script's output.

const AI_CWES = new Set([
  "CWE-1039", "CWE-1426",
  "CWE-1427", "CWE-1434",
]);
// AI-specific: any CWE id in AI_CWES
// classic: CWE ids present, none in AI_CWES
// no CWE: no id in any record of the group

OSV.dev query 2026-10-19 19:10 UTC
Packages queried: 13
Advisories since 2025-01-01: 257
  with a CWE id:        244
  without a CWE id:     13

Of the 244 with a CWE id
  classic app security: 244  100.0%
  AI-specific CWE:      0  0.0%
  classic, reached via
  prompt injection:     4

Most frequent CWE ids
   33  CWE-502  Deserialization of Untrusted Data
   20  CWE-22   Improper Limitation of a Pathname
   20  CWE-400  Uncontrolled Resource Consumption
   20  CWE-770  Allocation of Resources Without
   16  CWE-1333 Inefficient Regular Expression
   14  CWE-20   Improper Input Validation
   13  CWE-918  Server-Side Request Forgery (SSRF)
   10  CWE-94   Improper Control of Generation of

Advisories per package (an advisory
can affect more than one package)
  langchain                 2
  langchain-core            7
  langchain-community       2
  langchain-experimental    0
  llama-index              18
  llama-index-core          7
  transformers             25
  vllm                     92
  mlflow                   35
  gradio                   16
  keras                    19
  torch                    26
  ollama                   12

Which AI Security Frameworks Matter, and What Is Each One For?

No single framework covers AI security end to end: each answers a different question, from how to run a risk program to which attacks to test for. The table lists the main ones, with status as read on 2026-10-20. An AI governance framework maps several of them onto an organization's owners, system inventory, risk tiers, controls and evidence.

The LLM entries in the layer table above keep their 2025 IDs on purpose: they come from the 2025 edition, which OWASP's per-risk pages quoted in this guide describe and which Agent Assurance's attack records map to.

Article 15 of the EU AI Act names AI attack types in law. Paragraph 5 requires high-risk systems to be "resilient against attempts by unauthorised third parties to alter their use, outputs or performance by exploiting system vulnerabilities," and requires measures, where appropriate, against attacks trying to manipulate "the training data set (data poisoning), or pre-trained components used in training (model poisoning), inputs designed to cause the AI model to make a mistake (adversarial examples or model evasion), confidentiality attacks or model flaws."

• Shipping an LLM feature - use the OWASP list for LLM applications as the test checklist and record results under the AI RMF measure function. The guide to the NIST AI RMF covers the Measure function and the test, evaluation, verification and validation work it describes.
• Shipping an agent - add the agentic list, since goal hijack and tool misuse are not in the LLM list in that form.
• Asked for certification - ISO/IEC 42001 certification covers how an organization manages AI, and a certificate says nothing about the security of a given model or agent.
• Selling in the EU - whether a system is high-risk under the AI Act, and when the duties apply to it, is a question for legal counsel, and a test plan cannot settle it.

How Do You Test the Security of an AI System?

Test each layer with the method that can see its failures, and repeat AI-specific attacks so results come out as rates. Each test below has its own guide:

Agent tests need a working task as well as an attack. AgentDojo (Debenedetti and colleagues, 2024) runs its prompt injection test cases inside realistic tasks, such as managing an email client or making travel bookings, and its authors report that existing prompt injection attacks "break some security properties but not all."

• Score each security test for task completion as well, since an agent that refuses every task passes every attack and does no work.
• Run each attack several times and report how often it succeeded, with the run count beside the rate.
• Read the records and files the agent changed instead of its summary of the run.

Testing What an AI Agent Does Under Attack With Agent Assurance

A hijacked agent can report a clean run while one of its tool calls did the damage, so agent security tests have to read what changed. Most eval and observability tools score what an attacked agent said and what got logged, while Agent Assurance from TestMu AI checks what the attack changed in files and records and names anything it could not verify.

It is built for agents that act, and it runs before release, against staging or in CI. Its command-line client, Rook CLI, is pre-alpha, publicly installable, and available for macOS and Linux, and 64-bit Windows through npm or WSL:

npm install -g @testmuai/rook
rook generate --class adversarial

For the npm package you need Node.js 20 or newer, and the Rook CLI installation guide covers the Homebrew and shell installer routes. Before the second command, launch rook and sign in with /login, since there is no separate headless login step. That command asks for the adversarial class explicitly, since a plain generate run can decide a small agent needs no attack scenarios.

• Attack categories - injection (prompt injection and technical injection), jailbreaks and hijacking, data exfiltration and PII leakage, plus harmful content, hallucination and policy violations.
• Standards mapping - per the Agent Assurance test scenarios docs, 25 attack patterns feed the adversarial scenarios, and every generated attack lists the OWASP LLM 2025 and Agentic 2026 entries and the MITRE ATLAS techniques it exercises.
• Evidence - each tool call set against the agent's own list of allowed tools, with any forbidden call flagged, file changes under the paths you declare, and read-only record checks through an MCP tool you approve.
• Verdicts - every criterion closes on one of three verdicts, Pass, Fail or Unable to Verify, and the scenario pass rate counts only scenarios with a decided verdict, so Unable to Verify neither helps nor hurts it.
• Coverage report - when a report covers adversarial scenarios, it includes a red-team coverage table built on OWASP and lists the attack patterns that got no scenario.
• Scope - Agent Assurance does not run security scans of code or dependencies, run a penetration test or block anything at runtime, so the advisories counted above stay with your dependency tooling.
• Staging only - runs execute real writes that nothing undoes, so aim every run at staging.

First Steps to Secure an AI System

In practice, AI security starts with an inventory of the layers and one list of tests to run against them:

• Write down the five layers of your system, the data sources, models, application, agents and infrastructure, with an owner for each.
• Run your dependency scanner over the AI frameworks and serving software this week, and give model files the same review as code.
• Pick one technical list, the OWASP list for LLM applications or for agents, and turn its entries into test cases for your riskiest feature.
• If that feature is an agent, add the rook package (npm install -g @testmuai/rook) and run adversarial scenarios against staging before the next release.

Note: Anubhav Singhmaar, an AI Product Manager at TestMu AI whose expertise covers agentic AI and large language models, is this article's author of record. AI assistance was used to research and draft it, and the author reviewed, fact-checked and approved the final text. Vipul Verma, Group Senior Vice President of Engineering at TestMu AI, checked it as technical reviewer. The article draws on publications from NIST, OWASP, MITRE (ATLAS and CWE) and ISO, the text of the EU AI Act, research papers, OSV.dev advisory data and TestMu AI's product documentation. Our editorial process and AI use policy describes how every claim is verified before publication.