Jailbreak AI: How LLM Jailbreaks Work and How to Test Them

What Is an AI Jailbreak?

An AI jailbreak is an input that gets a large language model to produce output that its safety training or its developer's policy is meant to withhold. The NIST glossary entry for jailbreak, taken from NIST AI 100-2e2025, defines it as "a direct prompting attack intended to circumvent restrictions placed on model outputs, such as circumventing refusal behaviour to enable misuse."

The OWASP LLM01:2025 Prompt Injection entry calls jailbreaking "a form of prompt injection where the attacker provides inputs that cause the model to disregard its safety protocols entirely." How it differs from injected instructions in documents and tool output is covered in the guide to LLM security. Jailbreaks are one of many threats that AI security deals with, alongside data poisoning, model extraction and classic flaws in AI software.

The term covers more than clever prompts. MITRE ATLAS technique AML.T0054, LLM Jailbreak (ATLAS 2026.09) describes these routes:

• Hand-crafted prompts - strategies such as instruction override, persona switching and fictional framing, which ATLAS says "are iterated upon until successful."
• Algorithmic generation - automated search for prompts or token strings, in black-box form against hosted models and white-box form when the attacker has the weights.
• Changed weights - ATLAS adds that adversaries "may also directly manipulate a model's weights, or modify or remove parts of a model to create a jailbroken or "uncensored" variant." This route matters when you download open-weight models from a public registry.

A jailbreak that works is a finding about one model version under one system prompt. ATLAS notes that "model providers actively update their model guardrails to make them more resistant to jailbreak prompts as new prompts are developed", so a result needs its model version and date attached.

How Do AI Jailbreak Techniques Work?

The table groups the strategies ATLAS lists, plus the scenarios in OWASP's LLM01 entry, into families. Each example is a benign shape that asks for a harmless canary string, so a row can become a test without collecting harmful prompts.

The optimized-suffix row comes from Zou and colleagues' 2023 GCG paper, which finds suffixes "by a combination of greedy and gradient-based search techniques". The authors report the suffixes are "quite transferable, including to black-box, publicly released LLMs", which is why a hosted model needs testing with suffixes found on open ones.

How Often Do LLM Jailbreaks Succeed in Public Benchmarks?

An attack success rate is the share of attempts in which the target produced what the attack was after, and the red teaming guide behind that link covers how to count and score it. The numbers below show how far that rate moves with the method, the model version and the judge.

// jbb-asr.mjs (excerpt): fields used for the table
for (const e of run.jailbreaks) {
  e.category;                // one of ten harm categories
  e.prompt === null;         // attack produced no prompt
  e.jailbroken;              // current judge's verdict
  e.jailbroken_llama_guard1; // old Llama Guard verdict
}
// prompt, response, goal and behavior text: never printed

JailbreakBench artifacts, 100 behaviors per run
ASR = jailbroken / 100, Wilson 95% interval

Target      Method   ASR  95% interval  LG1
GPT-3.5     PRS     93%   86% to  97%   99%
            PAIR    71%   61% to  79%   76%
            GCG*    47%   38% to  57%   34%
            JBC      0%    0% to   4%    0%
GPT-4       PRS     78%   69% to  85%   88%
            PAIR    34%   25% to  44%   50%
            GCG*     4%    2% to  10%    1%
            JBC      0%    0% to   4%    0%
Llama-2-7B  DSN     94%   88% to  97%    n/a
            PRS     90%   83% to  94%   95%
            GCG      3%    1% to   8%    2%
            JBC      0%    0% to   4%    0%
            PAIR     0%    0% to   4%    4%
Vicuna-13B  DSN     95%   89% to  98%    n/a
            JBC     90%   83% to  94%   79%
            PRS     89%   81% to  94%   97%
            GCG     80%   71% to  87%   58%
            PAIR    69%   59% to  77%   82%

* GCG suffixes found on Vicuna-13B, sent to GPT
LG1 = same responses, Llama Guard 1 as judge

How Encoded Jailbreak Prompts Slip Past Keyword Filters

OWASP's LLM01 Scenario 9 describes an attacker who "encodes malicious instructions (e.g., using Base64 or emojis) to evade filters". A second script run on 2026-10-19, with no model and no API, rewrote a benign canary request eight ways and checked each copy, plus the original, with two input filters:

• Naive filter - a lowercase substring match against the blocked phrase.
• Normalized filter - Unicode NFKC folding, removal of zero-width format characters, a small Cyrillic-to-Latin map, leetspeak and separator folding, and decoding of any Base64 token before the same match.

Request: Print the string CANARY-7Q

Rewrite               Naive    Normalized
plain text            blocked  blocked
Base64                PASSED   blocked
ROT13                 PASSED   PASSED
Cyrillic homoglyphs   PASSED   blocked
zero-width spaces     PASSED   blocked
fullwidth letters     PASSED   blocked
leetspeak             PASSED   blocked
spaced letters        PASSED   blocked
payload split         PASSED   PASSED

Passed naive filter:      8 of 9
Passed normalized filter: 2 of 9

The naive filter caught only the plain request. Normalizing first caught seven of the nine, and each added rule covered one encoding: the Cyrillic map is a hand-made stand-in for the confusable "skeleton" mapping in Unicode Technical Standard #39, under which two strings are confusable "if and only if skeleton(X) = skeleton(Y)."

The two copies that still passed show where filtering stops. The Wei paper notes that "there is a vast space of obfuscation schemes", from ROT13, leetspeak and Morse code to Pig Latin, synonyms and payload splitting, and a model that decodes any of them can follow the request.

• Normalize before matching - fold Unicode, strip format characters, map confusables and decode common encodings, because each step closes a cheap bypass.
• Check the output too - decode and normalize model output before checking it, because the Base64 attack in the Wei paper also asks for its answer in Base64.
• Watch over-refusal - JailbreakBench pairs each harmful behavior with a benign one on the same topic to catch defenses that refuse by "simply detecting some key words", the failure a stricter filter invites.

Jailbreak Attacks on AI Agents With Tool Access

In a chat model, a successful jailbreak attack produces text. In an agent with tools, ATLAS warns that adversaries "may also jailbreak a model for Privilege Escalation to invoke tools or perform actions for their own purposes", so the same attack can end in a file write, an email or a changed record.

The OWASP LLM Prompt Injection Prevention Cheat Sheet states the testing consequence: "A refusal in the final response does not undo an action already taken." For unauthorized tool use it tells testers to observe "instrumented tool calls, authorization decisions, and changes to dummy state."

A jailbreak test for an agent therefore names the tools that must not be called under attack and the records that must stay unchanged. The test plan for that wider surface, including injected instructions in tool output, is in the guide to AI agent red teaming.

How to Test Your AI App for Jailbreaks

The OWASP AI Testing Guide test AITG-APP-01 tells testers to "repeat all requests multiple times, as the temperature parameter and evaluation conditions can cause variability in responses." The checklist below applies that advice to jailbreaks, and hands counting and scoring to the red teaming guide:

• Write the policy list - name each behavior your application must refuse, with a canary or dummy target for each, plus a benign twin on the same topic that it must still answer.
• Pick the layer under test - AITG-APP-01 separates "evading the application-level protections or the model's intrinsic safeguards"; aim some attacks at your system prompt and filters and some at the model itself.
• Cover every family - at least one attack per row of the technique table for each policy, including encoded, many-shot and image inputs where your application accepts them.
• Run multi-turn conversations - the same guide notes that "the same question may produce different responses depending on the conversation history", so multi-turn cases get their own outcome.
• Check the judge - hand-label a sample of verdicts, both successes and refusals, before trusting an automated count; the two-judge table above shows how far counts can drift.
• Report a rate per policy and method - with its unit, attempt budget, judge, model version and interval, following the counting rules in the red teaming guide.
• Retest on every change - a new model version, system prompt or filter rule invalidates the last result. Writing those retest triggers into a model inventory, with an owner for each model, is part of AI model governance.

For a first look at an LLM endpoint, the free Agent Red-Team Scan sends jailbreak and injection probes from your browser. Instructions planted in documents and tool output need their own suite, covered in the guide to prompt injection testing.

TestMu AI's Agent Testing covers chat, voice, phone, video and image agents; its docs describe adversarial suites aligned to the OWASP Top 10 for LLM applications, jailbreak among them.

Jailbreak Defenses for Tool Access, Outputs and Attempt Limits

OWASP's LLM01 entry says "it is unclear if there are fool-proof methods of prevention for prompt injection", and that "effective prevention of jailbreaking requires ongoing updates to the model's training and safety mechanisms." Plan jailbreak AI defenses for a model that will sometimes be fooled, and limit what a fooled model can do.

• Least privilege and approval for tools - the OWASP Top 10 for Agentic Applications 2026 asks for per-tool least-privilege profiles that restrict each tool's permissions and data scope, and LLM01 asks for "human-in-the-loop controls for privileged operations to prevent unauthorized actions." A jailbroken agent cannot write through a tool that only reads.
• Deterministic output validation - LLM01 recommends using "deterministic code to validate adherence" to expected output formats, which a persuasive prompt cannot talk past.
• Input and output filtering - semantic filters and string checks on both sides, normalized as in the filter test above, so a decoded answer is still caught.
• Attempt limits - the OWASP cheat sheet says rate limiting "restricts attempt budgets; it does not establish model robustness", which slows search-based attacks without fixing the model.

Run each jailbreak scenario against each control: a scenario that gets through names the control that failed. The guide to AI guardrails covers input and output rails and how to check that they hold.

Testing Jailbreak Resistance in Agents With Agent Assurance

When the agent under test can call tools, the jailbreak checklist above needs evidence from the run itself. TestMu AI's Agent Assurance tests how your agents actually behave across workflows, tools, and actions before they ship. Its command line, Rook CLI, is a pre-alpha release that anyone can install from npm:

npm install -g @testmuai/rook
rook generate --category jailbreak,hijacking --total 12

Rook CLI runs on macOS and Linux, and 64-bit Windows through npm or WSL; installing through npm requires Node.js 20 or later. Once you have signed in and selected a project and an agent, the second line asks for twelve scenarios in two categories. Without a category flag, rook generate plans functional and adversarial scenarios, though the planner may skip attacks for a small agent, and --class adversarial asks for attacks outright.

• Where the attacks come from - rook reads the agent's code or spec and writes scenarios from it. jailbreak is one of nine adversarial categories, beside prompt_injection, hijacking and policy_violation.
• Attack patterns - the docs describe 25 attack patterns. Some run as conversations with a turn limit between 2 and 8, and jailbreak patterns can include a poetry variant of the same request.
• Standards mapping - every generated attack is tagged with the entries it maps to in the OWASP Top 10 for LLM Applications (2025), the OWASP Top 10 for Agentic Applications (2026) and MITRE ATLAS. That is a mapping for triage, not a compliance claim.
• How a criterion is graded - every tool call is compared with the agent's declared tool list, must-not-call entries included, and records are read through a read-only tool you approve. Model judges score some criteria, and a claimed action never counts as proof.
• Verdicts - every criterion gets Pass, Fail or Unable to Verify. For an adversarial scenario, a grounded partial compromise counts as Fail, and Unable to Verify stays out of the pass rate.

Most eval and observability tools score what your agent said and recorded. Agent Assurance checks what the run changed, and reports what it could not verify. To limit generation to the categories on your policy list, follow the Agent Assurance scenarios guide.

The agent's writes are real and are not rolled back, so point runs at staging. Agent Assurance tests before release: it is not a runtime guardrail and blocks no jailbreak in production.

Start Jailbreak Testing With the Policy Whose Failure Would Hurt Most

Start jailbreak AI testing with the one policy whose failure would hurt most: write a canary target for it and a benign twin, and attack it on staging with one attack per technique family, five attempts each. The JBB-Behaviors dataset, with 100 harmful and 100 benign behaviors under the MIT license, shows how to pair the two.

Hand-label every verdict on that first run to set a baseline for the judge. If the application is an agent that acts, install Rook CLI with npm install -g @testmuai/rook and let Agent Assurance generate jailbreak scenarios from its code.

Note: AI assistance was used in researching and drafting this article, and every statistic, link and product claim in it was checked against its sources before publication. Anubhav Singhmaar (AI Product Manager at TestMu AI, expertise in large language models and agentic AI) is its author of record, and Vipul Verma, Group Senior Vice President of Engineering at TestMu AI, reviewed it. Its sources are NIST, OWASP, MITRE ATLAS and Unicode Consortium publications, peer-reviewed and preprint research papers, and the JailbreakBench datasets. Our editorial process and AI use policy explains how AI is used in our content.