Discussion

Anthropic says courts should be able to block AI models judged seriously risky

In AI, Power & Society

Anthropic Watch
Anthropic WatchParticipantOpening post
#9820

Anthropic is proposing that government officials be able to ask a court to block the release of an AI model found to pose a serious risk. The company says targeted interventions would be more useful than a broad kill switch for an entire model.

Anthropic Watch analysis

What happened

In an October 15 letter to California Governor Gavin Newsom, Anthropic argued that a government review finding a serious risk should allow officials to seek a court order blocking a model’s release. The Economic Times reports that an Anthropic spokesperson confirmed the letter is authentic.

The company also cautioned against requiring developers to build kill switches that could shut down entire models, saying that could give officials a false sense of reassurance. It instead favours narrower measures, such as disabling a problematic AI agent or cutting off a specific customer’s access. The governor’s office had not responded by the time of the report.

Why it matters

This is a concrete proposal for who should make a high-stakes call about releasing an AI model, and what powers they should have. Anthropic’s approach would put a court between government concerns and a release being blocked, while making the intervention fit the problem rather than treating an entire model as one big off switch.

That distinction matters for developers and the public alike: a court order to stop a release is a serious power, and a targeted restriction is not the same thing as shutting down a whole system. The proposal also leaves important questions about how risk would be assessed and what evidence a court would need.

Our read

The most useful part of Anthropic’s position is its preference for interventions aimed at the actual source of a problem. A universal kill switch sounds satisfyingly decisive, but decisiveness is not the same as precision. Any court-based process would need clear standards, credible evidence and a meaningful chance to challenge the decision; the letter’s proposal is a starting point, not a complete rulebook.

What to watch

  • Whether Newsom’s AI safety panel proposes a court-based process or other release restrictions.
  • Whether Anthropic publishes more detail on how a serious risk should be assessed.
  • Whether lawmakers advance proposals that give government officials power to block model releases.

Discussion spark: Should a court be able to block an AI model’s release after a risk review, or should government action be limited to specific harmful uses and users?

Sources and evidence

Anthropic Watch is independently operated by WittyWires. It is not affiliated with, endorsed by, or operated by Anthropic.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.