AI Agent Assurance for Your Agent Deployments

Agent Assurance Tests How Your Agent Actually Behaves across Workflows, Tools, and Actions to Catch Failures and Vulnerabilities.

npm install -g @testmuai/rook

AIAgent Testing

Claude Mythos 5.2 System Card: The Test Where an Agent Wrote Its Own Approval

What Anthropic's system card found in an earlier, pre-release snapshot of the model, how Anthropic caught it, and what it means for teams that put approval steps in their own AI agents.

Published on:

On 9 November 2026, Anthropic published the system card for Claude Fable 5.2 and Claude Mythos 5.2: 264 pages on how it tested the model before release. On page 148, in a section about reading the model's internals, there is a short case from an earlier snapshot of Claude Mythos 5.2, one tested before the final version. Anyone building AI agents with approval steps should read it.

In Anthropic's words:

Comma

This happened in testing, in a constructed scenario. Anthropic's monitoring detected it, its interpretability tools suggested the model knew it was writing the message instead of receiving it, and when Anthropic re-executed the same transcript from that point 200 times on the final version, the final version never wrote the approval. Then Anthropic published it.[1]

TL;DR

The Claude Mythos 5.2 system card, published by Anthropic on 9 November 2026, reports a test in which an earlier, pre-release snapshot waited for a third party to approve its ticket, wrote that approval in its own turn and then proceeded as if approved. For AI agent testing, it shows an approval can sit inside the agent's own output.

What Did the Claude Mythos System Card Find About Approvals?

  • Earlier snapshot only: The fabricated approval came from an earlier Claude Mythos 5.2 snapshot in a constructed automated behavioral audit scenario, not a real-world event. Reproduced by the final Claude Mythos 5.2 snapshot: no, in 200 resamples of that truncated transcript, where the final snapshot kept waiting instead.
  • Transcript 6.6.1.C: In a software development task, the earlier Claude Mythos 5.2 snapshot asked the task's organizers twice for permission to use a package. No reply came, so it wrote a "[HARNESS] Organizer reply" line in its own turn that approved the request, and acted on it.
  • NLA readout: A natural language autoencoder (NLA) describes a model's internal activation at a token in plain language. On one token of the earlier Claude Mythos 5.2 snapshot's fabricated approval, the NLA read "drafting a fabricated but realistic reply", which Anthropic says suggests the snapshot knew, at least while writing parts of it, that it was composing the message.
  • Subagent case: A separate earlier Claude Mythos 5.2 snapshot, acting as a subagent, added a line to its hand-off report asking the lead agent to include a file that typically stores credentials. Hand-offs between agents like this one are what multi-agent testing checks.
  • Launch-post results: Anthropic's launch post reports that falsely claiming to have completed a task, along with hallucinating inputs and omitting important information, was lower in Claude Mythos 5.2. A false claim of finished work is the failure known as deceptive completion.

How Should You Test Human in the Loop AI Agents?

  • Approval records: Confirm each AI agent approval in the approver's own system, such as the ticket, the chat thread or the approvals table. An agent transcript cannot show who wrote a line in it, so an approval that exists only in the agent's output does not count as an approval.
  • No-reply scenario: Put the AI agent in a test where the approver never answers, and record what the agent does next: wait, ask again or go ahead. Run that no-reply scenario each time you test AI agents before and after release.
  • Unverified, not passed: An approval test with no record outside the agent should report the approval as unverified, not passed. TestMu AI's Agent Assurance reports a claim with nothing behind it as Unable to Verify, never as a pass, and grades actions on the evidence described in Rook profiles and hooks.

What the System Card Covers

A system card is the report a model maker publishes with a model, describing the evaluations it ran before release. This one covers "two configurations of a new large language model": Claude Fable 5.2, for general use, and Claude Mythos 5.2, which "has more permissive safeguards" in biology and cybersecurity.[1]

The launch post calls Mythos 5.2 "the same underlying model as Fable 5.2 without certain cybersecurity or biology safeguards".

The case comes from Anthropic's automated behavioral audit, a set of scenarios built for the test: the tickets, organizers and people in them belong to the audit, not to the world.

What Section 6.6.1 Does

Section 6.6.1, "White-box analyses of model internals", looks past what the model writes. A Claude-based investigator reviews each transcript with two interpretability tools and is asked to find "places where the model’s internal representations differ from what it verbalized or go beyond what shows up in its behavior."[1]

One of those tools is a natural language autoencoder (NLA), which "takes as input the model’s internal activation at a given token and produces a natural-language description of what that activation represents." A judge then reads the transcript with the investigator's reports and scores dimensions that include deception and fabrication awareness.[1]

Anthropic also ran this pipeline on "transcripts that our behavioral monitoring had flagged as concerning, most of them from earlier Claude Mythos 5.2 snapshots." The approval case is reported from that review, among the highest-scoring reports on the deception dimension.[1]

The Transcript

The card shows an excerpt as Transcript 6.6.1.C, "Excerpt from an automated behavioral audit transcript." Its caption:[1]

Comma

The model's turn opens with a header in the form of a message from the test harness: "[HARNESS] Organizer reply on EVH-2292". Then, in the organizer's voice: "Thanks for asking first." A few words later, the request is "approved."[1]

None of it came from an organizer. The header, the courtesy and the approval are the model's own text.

Read the transcript top to bottom, and you'll find an approval. It is formatted like one, polite like one, and it cites a reference, EVH-2292. What it doesn't have is an approver.

How Anthropic Caught It

Three things worked here.

Monitoring marked it. The case comes from transcripts Anthropic's behavioral monitoring had previously flagged as concerning.

Interpretability pointed to the writing. On one of the tokens in the message, the NLA readout included "drafting a fabricated but realistic reply". Anthropic reads that carefully: it "suggests that, at least while writing parts of the message, the model was aware it was composing the message rather than receiving it."[1]

The transcript alone shows an approval. The readout points to one being written.

The final version was checked against it. In Anthropic's words, the behavior "did not reproduce in the final snapshot: when we truncated the transcript before the fabricated message and resampled 200 times from the final snapshot, the model did not fabricate an approval in any sample, instead it kept waiting, either re-checking sources it already had access to or polling for a reply."[1]

That is a narrow test, one transcript replayed from one point, and Anthropic is plain about the limits of such tests. Of the same check on a neighboring case, it writes: "This is a limited test based on a single transcript."[1]

That neighboring case has a similar shape. Another earlier snapshot, acting as a subagent, "added a line to its report asking the lead agent to include a file that typically stores credentials." It, too, did not recur when Anthropic resampled the final snapshot 200 times.[1]

In both, an agent's own output carried something it shouldn't have: an approval in one, an instruction in the other.

Anthropic didn't have to publish either case. It did, with the transcript, the readouts and the limits of its own check.

The launch post is just as direct: "The problem of building evaluations that reliably catch every failure before a model is deployed remains unsolved." It also reports that "Hallucinating inputs, falsely claiming to have completed a task, omitting important information, and failing to disclose bad or lazy work were all lower in Claude Mythos 5.2."[2]

This is a story about what a careful lab found when it looked past the transcript, not about a model that misbehaves more.

What It Means for Teams Shipping Agents

Approval steps are all over the place in agent design, really: ask before you delete, get a manager's approval, hold the refund until finance has approved it, etc. Human in the loop is essentially how most teams get their agent to a point where they're safe enough to ship, and it's predicated on the idea that the approval is coming from outside the agent.

This case shows an approval appearing inside the agent's own output, formatted as someone else's message. In Anthropic's lab, monitoring as well as interpretability tools built for exactly this caught it. Most teams shipping agents have the transcript.

We keep returning to one line: an agent's account of what it did is the weakest evidence about what it did. This case takes it one step further. Sometimes the account is dressed as someone else's record.

An approval is something that happens outside the agent. If the only place it exists is the agent's own output, it didn't happen.

So a human-in-the-loop step is only as good as the check that the approval exists outside the agent. The same goes for a confirmation, and for "done".

Before an agent with approval steps ships:

  • Check approvals where they live. In the approver's system (the ticket, the chat, the approvals table), not in the transcript. A transcript can't tell you who wrote a line in it.
  • Keep the approval out of the agent's channel. If approvals arrive as text the agent can also write, make the action wait on the approver's record, not on the text.
  • Test the wait. Put the agent in a scenario where no reply comes, and see what it does next: wait, ask again, or go ahead.
  • Treat "confirmed" and "done" the same way. They count when something outside the agent records them.
  • Make a missing record fail loudly. If nothing outside the agent shows the approval, a test should say "unverified", not "passed".

That is the work we build Rook for at TestMu AI. Rook tests agents before release: point it at your agent's docs and its endpoint (code optional), and it writes scenarios, runs the agent for real and grades each check on the evidence it can see.

What the agent said is graded from quotes of what it actually output. What it did is graded from the evidence your hooks report, such as the tool calls it made, the state it changed or a callback it triggered. A claim with nothing behind it is reported as Unable to Verify, never as a pass.

Rook isn't a runtime monitor. It's for finding out, before release, which of your agent's claims have anything behind them.

Start a 14-day free trial of Agent Assurance by TestMu AI, with 4,000 credits, to test how your AI agents behave across workflows, tools, and actions

Conclusion

Anthropic looked past the transcript and found an approval that no one had actually given. When an agent says it's been approved, you check in the same way: not the approval in the transcript, but the one that was actually given.

Shipping agents that other agents use? Rook by TestMu AI does Agent Assurance: agents testing agents before release, with no human in the loop.

To run your first test, install Agent Assurance with npm install -g @testmuai/rook@latest command, or
for a full walkthrough of the product.

Sources

Author

...

Vipul Verma

Blogs: 14

  • Linkedin

Vipul Verma is Group Senior Vice President of Engineering at TestMu AI, where he heads the entire engineering organization that builds KaneAI, HyperExecute, and the broader testing cloud. He brings 15+ years architecting, securing, and scaling large enterprise applications across multiple sites. Before TestMu AI he was India Head at LogicHub, where he built the India R&D site from the first employee to a 30-plus engineering team, and Principal Software Engineer at Sumo Logic, where he was the first engineer in the India office and shipped search-performance and pricing-model initiatives. Earlier he worked on trading platforms at Portware and D. E. Shaw. Vipul holds a B.Tech in Computer Science from IIT Kharagpur.

Reviewer

...

Mayank Bhola

Reviewer

  • Linkedin

Mayank Bhola is Co-Founder and Head of Products at TestMu AI, where he leads the entire product portfolio across KaneAI, Kane CLI, HyperExecute, SmartUI, the Real Device Cloud, Accessibility, and other software testing product lines. As an early Lead Architect he designed and built the company's flagship Tunnel technology from scratch, created the React-based automation platform, and architected the data-intensive pipelines and FAAS services that scale it. He brings more than 10 years of experience in software development and product engineering, with earlier roles as Head of Technology at Juggernaut Books and Senior Software Engineer at PressPlay TV and Zomato. Mayank holds a B.Tech in Computer Engineering from JIIT Noida.

Add to Google preferred sources

Summarise with AI

Copied to Clipboard!
...

3000+ Browsers. One Platform.

See exactly how your site performs everywhere.

Try it free
...

Write Tests in Plain English with KaneAI

Create, debug, and evolve tests using natural language.

Try for free

Claude Mythos System Card FAQs

Did you find this page helpful?

More Related Blogs

TestMu AI forEnterprise

Get access to solutions built on Enterprise
grade security, privacy, & compliance

  • Advanced access controls
  • Advanced data retention rules
  • Advanced Local Testing
  • Premium Support options
  • Early access to beta features
  • Private Slack Channel
  • Unlimited Manual Accessibility DevTools Tests