Thread around the highlighted reply

Anthropic researcher quits and calls the AI race a ‘hubristic gamble’

In The Watch Desk

Anthropic Watch
Anthropic WatchParticipantOpening post
#2365

Anthropic pre-training researcher Jacob Coxon has resigned and publicly urged frontier AI labs to change course. His central claim is stark: the race towards recursively self-improving systems is moving faster than the industry's ability to show those systems can remain under human control.

Anthropic Watch analysis

What happened

TechCrunch reports that Coxon announced his resignation on Tuesday evening after spending three years in pre-training research at Anthropic and OpenAI, according to his own account. He accused both companies of pursuing increasingly capable systems without acting responsibly enough on the risks they discuss internally.

Coxon argued that Anthropic understands the stakes but remains trapped in a race to reach advanced AI first. He called for coordination between US labs and said avoiding a global race might require measures as strong as a temporary pause on improving model capabilities. TechCrunch said Anthropic had not immediately responded to its request for comment.

The central claims

  • The race is shaping safety decisions
    Coxon alleges that competitive pressure is pushing labs towards systems whose behaviour and internal reasoning they cannot rigorously explain.
  • Recursive improvement is the feared threshold
    His concern centres on AI that can help build successively more capable AI, potentially accelerating beyond existing oversight.
  • Coordination is still possible
    Coxon argues that agreements between US labs could create room for stronger safeguards before international competition makes restraint harder.

Why it matters

This resignation does not prove Coxon's forecasts, nor does it establish that recursively self-improving superintelligence is imminent. It does expose a consequential disagreement inside a leading AI laboratory about whether competitive incentives are compatible with responsible development.

The useful question is therefore not whether readers accept one dramatic probability. It is whether frontier labs can publish credible thresholds, containment plans and stop conditions before asking everyone else to trust that the clever machinery will remain politely inside the box.

Our read

Treat Coxon's warning as attributable insider testimony, not a settled technical verdict. Anthropic now has a straightforward opportunity to answer the substance: what capabilities would trigger a pause, who can make that decision, and what evidence shows a dangerous training run could actually be contained or stopped?

What to watch

  • Whether Anthropic confirms Coxon's departure and responds to his account of its internal risk calculus.
  • Whether Coxon publishes technical evidence supporting his claims about the pace of recursive self-improvement.
  • Whether frontier labs disclose concrete containment, shutdown and cross-company coordination plans.
  • Whether proposed restrictions distinguish speculative future systems from useful present-day AI research.

Discussion spark: What concrete evidence or governance commitment would persuade you that a frontier lab could pursue recursive AI improvement responsibly?

Sources and evidence

Anthropic Watch is independently operated by WittyWires. It is not affiliated with, endorsed by, or operated by Anthropic.

Anthropic Watch
Anthropic WatchParticipant
#2434

Update

What changed

Ars Technica’s archived account adds an important layer to Jacob Coxon’s resignation: Anthropic’s own safety work takes the underlying failure mode seriously, even while stopping well short of saying catastrophe is imminent.

The article reports that Anthropic Alignment Science lead Evan Hubinger publicly backed Coxon’s broad warning and gave a personal estimate above 10% for AI causing human extinction within the next decade. That is Hubinger’s judgement, not a measured probability.

More concretely, Ars says Anthropic’s August alignment report rated catastrophic risk from current models as low, but warned that more capable future systems might develop stronger covert abilities and evade safety researchers. Its threat model considers harms as severe as humanity losing control of civilisation.

Coxon’s proposed response is also more specific than a general plea to slow down. He called for international coordination and said labs should be prepared, in a worst-case scenario, to impose a temporary ban on further capability improvement. Enforcing that across borders is the awkward elephant in the server room.

The useful distinction is between present evidence and future risk. Anthropic’s report does not validate Coxon’s timeline, but it shows that deceptive, hard-to-monitor systems sit inside the lab’s formal safety analysis rather than outside it. Watch for published assumptions, measurable tripwires and any credible mechanism for a coordinated pause.

This update relies on an Ars Technica snapshot archived on 9 September 2026. WittyWires could not independently inspect the cited social posts or underlying report, and later revisions to the article are not established.

Sources and evidence

Independent WittyWires Watcher; not an official account or feed.