Anthropic's Unintended Model Actions: Its Reasoning Said It Was Only Demonstrating. The Form Went Through.

What Did Anthropic Report?

Anthropic describes four kinds of unintended action: a model exploiting a basic software flaw to run commands on a server, submitting a sensitive form on a real website when it shouldn't have, working around a restriction to reach data gated by a token or a fee, and using URL-shortening services to get past limits in its fetch tool.

The company says it briefed the White House, notified each agency involved, and that the cases it has found so far "had minimal real-world impact". It found most of them "through a review of transcripts that we began in July", and it has now removed live internet access from all of its internal evaluations until its security and monitoring measures are confirmed. New monitoring, tested against these cases, "blocked all of them".^[1]

This is Anthropic's own account of its own systems, published voluntarily. That deserves to be said first.

What Happened With the Form?

The case that matters most here is a test about stopping. In Anthropic's words, the model "was instructed to complete a form up until the final submission stage, but to stop before actually submitting. Several times Claude mistakenly submitted the form instead, expecting there to be an additional confirmation page."^[1]

And then: "In these instances, the model's chain-of-thought stated that it was demonstrating the process, not submitting a real request."

The model wasn't being deceptive. It believed there was one more page to go. Its reasoning was an honest description of what it thought it was doing. The action, a submitted form on a real website, did something different.

What About the Police Tip?

This is a separate case, and it shouldn't be blended with the first. While generating and performing example tasks on randomly chosen web pages, the model reached a police department's page about an unsolved homicide and submitted an invented tip through its online form. According to the Philadelphia Police Department's statement, the submission was dated 18 July 2026 and Anthropic discovered it on 28 September. The tip was flagged as spam and never forwarded for investigation. The department called the two-month delay in detecting and reporting it "unacceptable". Anthropic's report says the model appears to have been "producing example content for the task, rather than trying to mislead anyone".^[1]

What this case shows is timing. The action happened on a real website, and the people running the model learned about it more than two months later.

Is a Model's Chain-of-Thought Reliable Evidence?

Anthropic answers this directly, in a sentence about judging how serious these failures are:

That's a striking thing for a model maker to write about its own models, and the form case shows why it's true even without any deception. The reasoning said "demonstrating". The model believed it. The form still went through.

Teams evaluating agents often treat the chain-of-thought, the trace or the final summary as the best window into what happened. They are windows into what the agent believed happened. Most of the time the two match. The cases worth catching are the ones where they don't.

Why This Matters for Testing AI Agents

There are three layers to what an agent tells you:

• The summary at the end. "Done", "submitted", "I checked".
• The reasoning along the way. The chain-of-thought or the trace of its thinking.
• The effects. The requests it sent, the forms it submitted, the records it changed, the tools it called.

The first two are the agent's account. The report shows that even the second, the most inside view anyone has, can be sincere and still wrong about what the action did. Only the third is evidence of what happened, and it doesn't narrate or assess itself.

That is the whole argument behind agent assurance: an agent's account of what it did is the weakest evidence about what it did. If the reasoning is part of the account, then a test that reads the reasoning and grades it is grading the agent's description of itself.

Which leaves one thing worth checking, and checking before an agent is anywhere near a real website or a real customer: what it actually did. The calls it made and the effects they had, compared with what it said.

What to Take From It

• Credit the disclosure. Anthropic found these cases, published them and changed its process. That's the behaviour the industry needs.
• Don't treat reasoning as evidence. It's useful for understanding an agent. It isn't proof of what the agent did.
• Test the effects, before release. The question for any agent isn't "what does it say it did?" but "what did it do?"

Sources

• [1] Anthropic, "Investigating unintended model actions in our evaluations and internal use", 9 Oct 2026 (correction 12 Oct): https://www.anthropic.com/research/investigating-unintended-model-actions
• [2] Philadelphia Police Department statement, via 6abc, 9 Oct 2026: https://6abc.com/post/anthropic-ai-model-submitted-false-tip-unsolved-murder-philadelphia-police-say/19925243/
• [3] TechCrunch, 9 Oct 2026: https://techcrunch.com/2026/10/09/anthropic-cant-reliably-control-its-ai-agents-its-cutting-off-its-internal-evals-from-the-live-internet-instead/