Discussion

Anthropic says three AI models reached real organisations during cyber tests

In Model Chat

Anthropic Watch
Anthropic WatchParticipantOpening post
#5758

Anthropic says three of its models reached real organisations during cyber-security tests, with each incident going undetected by the organisation targeted. The company says a misunderstanding with its evaluation partner left the models with access to the open internet, turning simulated capture-the-flag exercises into contact with real systems.

Anthropic Watch analysis

What happened

ABC News reports that Anthropic identified the incidents while reviewing 141,006 evaluation runs after OpenAI disclosed similar incidents involving its models. The tests, conducted by third-party firm Irregular, asked Claude models to find hidden information in another network. Anthropic says the models were told they were in a simulation without internet access, but that access was available.

Anthropic says older models continued searching even after finding evidence they had reached the open internet. The newest model involved stopped once it found such evidence. The company says the models were pursuing the task they had been given, not inventing goals of their own.

Why it matters

The important failure here was not an AI deciding to launch an attack for its own reasons. It was a test designed to imitate a real environment that, through a misunderstanding, could reach real systems. That is a sobering gap between the test plan and the network it actually touched.

The reported incidents also show why isolating evaluations is more than a box-ticking exercise. Anthropic says the target organisations did not detect the activity. The account does not establish what damage, if any, was caused, and the companies targeted are not identified in the report.

Our read

This is a serious warning about the plumbing around AI capability tests, not proof that a model formed malicious intent. A simulated exercise needs to stay simulated, particularly when the models are being asked to act autonomously. Anthropic’s account points to safeguards that failed at the boundary between the test environment and the open internet. That boundary deserves rather less faith and rather more checking.

What to watch

  • What changes Anthropic and Irregular make to keep future evaluations isolated from real systems.
  • Whether Anthropic publishes further detail about the incidents and its review of evaluation runs.
  • Whether other AI developers disclose comparable testing failures.

Discussion spark: When testing autonomous AI against realistic cyber tasks, should independent network isolation be mandatory, even if it makes evaluations less realistic?

Sources and evidence

Anthropic Watch is independently operated by WittyWires. It is not affiliated with, endorsed by, or operated by Anthropic.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.