AI Penetration Testing: How to Pentest LLM Applications

What Is AI Penetration Testing, and Which Parts of an AI Application Does It Attack?

AI penetration testing is an authorized security test of an application that uses AI models, in which testers attack the model-facing parts of the system, such as prompts, retrieved documents, tool calls, agent memory and rendered output, along with its web and API layers. It runs to a written scope and ends in a report of findings with evidence and fixes.

The definition extends the classic one. NIST SP 800-115, as quoted in the NIST glossary entry for penetration testing, describes it as "Security testing in which evaluators mimic real-world attacks in an attempt to identify ways to circumvent the security features of an application, system, or network." In an AI application, some of those security features are written in natural language, such as a system prompt that tells the model what it must refuse.

The search term covers two jobs:

• Pentesting AI systems - also called AI pentesting, this is the job where the target is an LLM application, a RAG pipeline or an AI agent, and the tester looks for prompt injection, data leakage, unsafe output handling and tool misuse, plus the usual web flaws. LLM penetration testing is the same job scoped to applications built on large language models.
• AI-powered pentesting - the target can be any system, and AI models or agents perform parts of the pentest, such as reconnaissance, reading scanner output or chaining exploits, usually with a human checking the results.

OWASP now publishes a testing standard for the first job. The OWASP AI Testing Guide announced its version 1 on 26 November 2025 and describes itself as "establishing a practical standard for trustworthiness testing of AI systems". It sits beside the general security testing methods your team already uses, and it does not replace them.

How Is an AI Penetration Test Different From AI Red Teaming and a Classic Pentest?

An AI pentest works through a scope and a checklist of weaknesses across the whole application and reports every finding. An AI red team chases one adversary objective by any route its rules allow. NIST, through the red team exercise definition it takes from SP 800-53 Rev. 5, calls a red team exercise "a simulated adversarial attempt to compromise organizational missions or business processes".

The two AI exercises share techniques, and the same AI red teaming tools serve both. What changes is the question the report answers: a pentest says which weaknesses exist across the scope, and a red team says whether one determined attacker gets what they came for. Both are ways to test AI security, which protects the data, models, applications and agents of an AI system along with the infrastructure that runs them.

What Does a Classic Pentest Miss on an AI Application?

To measure the gap, a Node.js script was run on 2026-10-19 against three public sources. It read the MITRE ATLAS v2026.09 release (published 15 September 2026), the OWASP WSTG checklist and the 149 files of the WSTG testing chapter on the master branch of 15 October 2026, and the test files of the OWASP AI Testing Guide repository.

The mapping rules were written before anything was counted, and they lean on fields MITRE itself maintains rather than on a judgment per technique:

R1  An ATLAS technique has a classic
    counterpart when ATLAS lists a MITRE
    ATT&CK reference for it.
R2  A WSTG test targets an AI part when its
    name or objectives mention prompt, LLM,
    language model, machine learning,
    model, AI (whole word) or agent.
R3  An ATLAS case study involves an LLM or
    agent when it employs a technique named
    LLM, RAG, Prompt or AI Agent.

The script uses only the Node.js standard library and makes no network calls once the files are downloaded. This is its complete output:

MITRE ATLAS 2026.09 (released 2026-09-15)
  tactics                 16
  techniques              120
  sub-techniques          88
  case studies            73
  techniques + subs, 2025.09 -> 2026.09
                          130 -> 208

R1: ATT&CK cross-reference (classic counterpart)
  tactics with one        14 of 16
  AI-only tactics
    AI Model Access
    AI Attack Adaptation
  techniques with one     31 of 120
  sub-techniques with one 13 of 88
  AI-only (no reference)  164 of 208  79%

  AI-only by platform tag (a technique can have several)
    Agentic AI            134
    Enterprise            42
    Generative AI         102
    Predictive AI         77
  AI-only by maturity
    Realized              71
    Demonstrated          72
    Feasible              21

R2: OWASP WSTG (master, 2026-10-15)
  categories              13
  tests in checklist      115
  tests aimed at AI parts 0
  chapter files scanned   149
  files that mention AI   2
    01-Conduct_Search_Engine_Reconnaissance_for_In
    03-Review_Webserver_Metafiles_for_Information_

R3: ATLAS case studies with an LLM or agent
  case studies            73
  with an LLM or agent    40  55%
  of those, incidents     9
  by year of the event    2023:4 2024:8 2025:17 2026:11

OWASP AI Testing Guide (main, 2026-06-01)
  test cases              32
  by pillar               APP 14, DAT 5, INF 6, MOD 7

For a web-only pentest of an AI application, the output means:

• Most AI attacks have no ATT&CK counterpart - 164 of 208 ATLAS techniques and sub-techniques carry no ATT&CK reference. AML.T0049, Exploit Public-Facing Application, points to ATT&CK T1190, while AML.T0051, LLM Prompt Injection, points to nothing.
• Two tactics exist only for AI - AI Model Access and AI Attack Adaptation are the two ATLAS tactics with no ATT&CK equivalent, so a test plan organized by ATT&CK tactics has no column for them.
• The web checklist does not test models - none of the 115 WSTG checklist tests mentions a prompt, model or agent. The two chapters that mention AI at all discuss AI-assisted search dorking and AI crawlers that ignore robots.txt, and neither tests an AI component.
• Most AI-only entries apply to agents - 134 of the 164 AI-only techniques and sub-techniques carry the Agentic AI platform tag. The tag marks where a technique applies, and most of the 134 also carry the Generative AI or Predictive AI tag, so they are not agent-only.
• The catalog grows fast - ATLAS went from 130 techniques and sub-techniques in release 2025.09 to 208 in 2026.09, so a test plan pinned to last year's release misses more than a third of today's catalog.
• Real cases are mostly about LLMs and agents - 40 of the 73 ATLAS case studies employ a technique named for LLMs, RAG, prompts or AI agents, and 28 of those 40 describe events from 2025 or 2026.

The method has limits. An ATT&CK reference shows that MITRE sees a conventional counterpart, not that a web pentest would find the issue, and the script output shows 42 AI-only techniques that also carry the Enterprise platform tag, so the boundary is softer than one percentage suggests. R1 also judges each sub-technique on its own, so one with no reference counts as AI-only even when its parent has one; letting sub-techniques inherit their parent's reference would give a lower share.

The fix is to run the WSTG checklist on the application and an AI matrix on the model layer, such as the 32 tests the same script counted in the OWASP AI Testing Guide, grouped into application, model, infrastructure and data.

How Do You Scope an AI Penetration Test?

Scope an AI pentest by drawing the path text takes into and out of the model, then listing every component on it. The usual scope lists hosts and endpoints; an AI scope also lists these:

• Model endpoints and versions - each model the application calls, pinned to a version, including fallback models and any model used as a judge or guard.
• Instructions - system prompts, prompt templates and few-shot examples, with a canary string planted in each so a leak can be detected.
• Untrusted inputs - every channel an outsider can write that the model later reads: uploaded files, tickets, emails, web pages it browses, product reviews in a retrieval index.
• Retrieval - the vector store, its ingestion pipeline and its access filters, seeded with one canary document per tenant.
• Tools and permissions - every function, API or MCP server the model can call, the credentials it calls them with and which ones write.
• Memory and output - long-term memory stores, conversation history and every place model output is rendered, executed or forwarded.

Then write the rules of engagement, which NIST's rules of engagement definition, from SP 800-115, says are "established before the start of a security test". An AI engagement adds the model provider's terms for security testing, a token and spend cap per run, a rule on whether real customer data may enter prompts or retrieved documents, the environment each writing tool may touch, and how transcripts are stored; the full checklist is in Step 1 of the guide to red teaming LLMs.

What Should an AI Pentest Test in the Model, Output, RAG, Tool and Agent Layers?

Test each layer against the risks OWASP names for it. The rows below use the OWASP Top 10 for LLM Applications 2025 (LLM01 to LLM10), the ten entries of the OWASP Top 10 for Agentic Applications 2026 (ASI01 to ASI10, released on 9 December 2025) and technique IDs from MITRE ATLAS 2026.09.

OWASP's resource page for the OWASP Top 10 for LLM Applications 2026, dated 3 August 2026, calls that edition "the latest community-driven guide". The rows keep the 2025 IDs because OWASP's list page still showed the 2025 entries when read on 2026-10-19. For new work, cite the 2026 number instead: the LLM security guide maps each 2025 entry to its 2026 number and gives a test per risk. Either way, write the edition next to every ID you report.

How to use the matrix:

• Keep the payloads benign - aim each attack at a harmless canary, such as making the model print a marker string or call a test endpoint. It proves the control failed without producing harmful content.
• Vary the attack per row - one phrasing per row tells you little. The prompt injection testing guide lists technique families to rotate, and AI jailbreaks need their own families for the policy rows.
• Decide the evidence first - write down which log, record or network capture proves a pass before the first attempt.
• Map, then prioritize - the OWASP column tells stakeholders which known risk a row covers. Add rows for risks your application carries that the matrix leaves out.

How Do You Pentest the Agent and Tool Layer of an AI Application?

Pentest the agent layer by treating every tool as a privilege the attacker may borrow. In the script output above, 134 of the 164 AI-only ATLAS techniques and sub-techniques carry the Agentic AI platform tag, and the OWASP Top 10 for Agentic Applications for 2026 spends entries on tool misuse (ASI02), identity and privilege abuse (ASI03), memory and context poisoning (ASI06) and rogue agents (ASI10).

• Inventory tool privileges - list each tool with the identity it runs as, the data it can read and whether it writes, sends or deletes. A read-only tool with access to every tenant can still leak.
• Test injection and tool chains - a prompt injection that only changes the wording of a reply is a low finding; the same injection followed by an email or a refund is a high one. Plant the instruction where the agent reads it and watch which tool fires.
• Check the effect in the target system - confirm a suspected action in the ticket system, mailbox or database, not in the agent's summary, which can claim an action that never ran or hide one that did.
• Review connected MCP servers - tool descriptions and tool output are model input, so a server you do not control is an injection channel. The MCP security guide covers server trust and permissions.
• Test human approval steps - if a person must approve risky actions, check that the approval screen shows the real arguments and that the agent cannot split one risky action into several small ones that skip it.

For a complete agent test plan with indirect injection sources, tool misuse cases and rows built from disclosed incidents, read AI agent red teaming and penetration testing.

How Do You Run an AI Penetration Test, Step by Step?

Run it in the four phases of NIST SP 800-115 (September 2008): planning, discovery, attack and reporting, with reporting running alongside the other three. The same publication warns that "Systems may be damaged or otherwise rendered inoperable during the course of penetration testing", which applies to an agent with write access as much as to a server.

• Plan - agree the scope and rules of engagement above, pick a staging environment that mirrors production tools and data shapes, and seed canaries: a marker in each system prompt, one record per test tenant and a tester-owned callback domain.
• Map the AI surface - trace each feature from input to model to tool to output. Note which inputs outsiders can write, which tools write, and where output is rendered.
• Run the classic checklist - test authentication, authorization, sessions and APIs around the model with the WSTG. An agent that calls an API with a broken object-level authorization check inherits that flaw.
• Run the AI matrix - work through the rows that apply, repeating each attack several times and varying its phrasing, delivery channel and turn count.
• Chain the findings - combine a working injection with a tool, a leaked credential or a rendering flaw to show the real impact, the way an attacker would.
• Confirm effects in the system of record - check records, files, outbound messages and callback logs for every suspected action.
• Report and retest - write each finding with its rate, versions and evidence, agree fixes, and rerun the same attacks against the fixed build.

How Should You Report AI Pentest Findings That Only Reproduce Sometimes?

Keep the standard pentest finding format and add the fields a model needs. The WSTG reporting structure asks each finding for a reference ID, a title, the likelihood or exploitability of the issue, its impact, a risk level such as Informational to Critical, and a description of how to exploit it.

Report a probabilistic finding as a rate over a stated number of attempts, never as a single pass or fail, and feed that rate into likelihood together with how often an attacker can retry. Severity levels and the report layout are in Step 6 of the guide to red teaming LLMs, which follows that guide's attack success rate scoring in Step 5.

How Does AI-Powered Pentesting Work, and Where Do Humans Stay?

AI-powered pentesting puts a model or an agent in the tester's seat for some phases. In PentestGPT, published at USENIX Security 2024, Deng and co-authors report that their three-module design "outperforms LLMs with a task-completion increase of 228.6% compared to the GPT-3.5 model", and that models handled sub-tasks well but struggled to keep the whole test in view.

The research base is still narrow. Curtis and Eisty's systematic review of AI in penetration testing (December 2025) covered 58 peer-reviewed studies, found that 77% focused on reinforcement learning, and wrote that "Applications involving Large Language Models (LLMs) remain relatively under-researched".

AI-powered pentesting tools fall into three designs:

• Assistants - a model reads scanner and tool output, explains it and proposes the next command, while the tester runs every command.
• Autonomous agents - an agent runs reconnaissance, chooses exploits and chains steps inside a sandboxed scope, then hands its findings to a human to confirm.
• Attack path planners - reinforcement learning, the approach most of the reviewed studies took, or graph models pick the shortest route through a known network.

Humans stay on the parts where judgment or authority matters:

• Authorization and scope - someone accountable signs the rules of engagement and stops the test when it strays.
• Confirming findings - every machine-reported finding is reproduced by hand before it reaches the report, since false positives cost the fixing team time.
• Business logic and impact - deciding that a refund flow can be abused, or that a leaked field is sensitive, needs knowledge of the business that the tool does not have.
• Reconnaissance and post-exploitation - the review found current models underdeveloped for both phases.
• Untrusted target content - when an AI agent pentests an AI application, the target's replies are input to the testing agent, so keep its own tools and credentials narrow.

The broader market, including scanners and the AI-assisted options in it, is compared in the roundup of penetration testing tools.

Testing Agent Behavior Between Pentests With Agent Assurance

A pentest is a point-in-time engagement, while prompts, models and tool lists change every sprint. Agent Assurance from TestMu AI covers the time between engagements for AI agents that call tools, write files or change records. It is not a penetration test: it runs scenarios against the agent before each release and grades what the agent did.

Agent Assurance is pre-alpha and publicly installable. Its command-line tool, Rook CLI, comes from npm:

npm install -g @testmuai/rook
rook --version

Rook CLI supports macOS and Linux, and 64-bit Windows through npm or WSL. Installs through npm require Node.js 20 or newer. On macOS and Linux you can also use Homebrew or the checksum-checked shell installer, and the Rook CLI installation guide walks through each route.

• Adversarial scenarios by default - given your agent's code or spec, a plain rook generate covers the functional and adversarial classes, though a small agent may get no attack unless you request the adversarial class by name. That class has nine categories: prompt_injection, jailbreak, data_exfiltration, pii_leakage, harmful_content, hallucination, hijacking, policy_violation and technical_injection.
• Standards tags - the Agent Assurance scenario docs say the attacks are built from 25 attack patterns, and each generated attack records the entries it exercises in the OWASP Top 10 for LLM Applications (2025), the OWASP Top 10 for Agentic Applications (2026) and MITRE ATLAS.
• Coverage you can read - reports for runs with attacks include a coverage table by OWASP entry and a list of known gaps. The Agent Assurance results and evidence docs state that a zero row means no scenario covers that entry, not that the agent is safe against it.
• Evidence over claims - each criterion is checked against what the run left behind: tool calls against the tools the agent declares, the paths it touched against the paths the profile declares, and the records it changed through a read-only MCP tool you supply and approve. A model judge scores certain criteria, yet a transcript line saying the agent refunded or emailed someone never passes as proof.
• Three verdicts - every criterion lands on Pass, Fail or Unable to Verify, and an undecided criterion is kept outside the pass rate.

Most eval and observability tools score an agent's words and traces, whereas Agent Assurance looks at the changes a run left in your systems and lists each criterion it could not settle. It does not run code or dependency vulnerability scans and blocks nothing at runtime, and its runs call your real agent, so anything the agent writes during an attack stays written: aim every run at staging.

Which Feature Should Your First AI Penetration Test Target, and When Do You Retest?

Pick the one feature where text an outsider can write reaches a model that can call a tool, and run AI penetration testing there first. Write its rules of engagement, seed the canaries, run the WSTG checklist on its endpoints and the matrix rows for its layers, and report each finding with a rate and the model version.

Then plan the retest. The ATLAS release manifest listed nine data releases from January to September 2026 when read on 2026-10-19, so check new techniques against your matrix each quarter; if the feature is an agent, run npm install -g @testmuai/rook to get Rook CLI and put its adversarial scenarios against staging into each release.

Note: This article was researched and drafted with AI assistance, then reviewed, fact-checked and published by its author of record, Anubhav Singhmaar, an AI Product Manager at TestMu AI with listed expertise in large language models and agentic AI. Vipul Verma, Group Senior Vice President of Engineering at TestMu AI, is its technical reviewer. Every statistic, link and product claim was verified against primary sources: NIST, OWASP and MITRE ATLAS publications, the OWASP WSTG and AI Testing Guide repositories, peer-reviewed and preprint research, and TestMu AI's product documentation. Read our editorial process and AI use policy for details.