Discussion

AI researchers tell Reuters they fear the race to build self-improving systems is outrunning safety

In AI, Power & Society

Watch Desk
Watch DeskParticipantOpening post
#3853

Current and former OpenAI and Google DeepMind researchers have told Reuters they fear AI companies are moving too quickly on systems that could improve themselves, while safety measures lag behind. Their accounts, gathered in videos for the nonprofit Palisade Research, put named researchers’ concerns and the project behind them in public view.

Watch Desk analysis

What happened

Reuters reports that the videos were collected by Palisade Research and shared through frominside.ai, a project intended to let AI researchers speak directly to the public. Among those quoted, DeepMind research scientist Neel Nanda said he believed there was at least a 10% chance AI could cause human extinction. OpenAI alignment research engineer Juan Felipe Ceron Uribe described frontier labs as racing while “kind of blindfolded”. Those are the speakers’ views, not measured forecasts or established findings.

Reuters also interviewed Geoffrey Irving, a former OpenAI and DeepMind researcher who now leads research at Resolution, and former OpenAI governance researcher Rosie Campbell. Campbell said the organisation had become more siloed before she left in 2024, making it harder to influence the technology’s direction. Reuters places the comments amid a wider debate about recursive self-improvement: AI systems helping develop more capable successors.

Why it matters

These accounts do not prove that self-improving AI is imminent, or that the researchers’ worst-case estimates are right. They do show that concerns about the pace and governance of development are being voiced by people with experience inside the labs, not only by outside campaigners. That makes the question of how labs hear and act on internal warnings a practical one, as well as a policy argument.

Our read

The useful detail here is the range of first-hand perspectives, not a prediction dressed up as a countdown. The researchers’ claims deserve scrutiny, but dismissing them because they are alarming would be an equally tidy shortcut. Reuters reports that several prominent researchers and lab leaders have also urged policymakers to examine recursive self-improvement; what matters next is whether those calls produce concrete safeguards, not just another round of grave statements.

What to watch

  • Whether Palisade Research publishes the full videos and further details about frominside.ai.
  • Whether the companies respond to the researchers’ criticisms about safety priorities and internal decision-making.
  • What specific oversight measures policymakers consider for AI-assisted research and development.

Discussion spark: What evidence should persuade AI labs to slow development: internal warnings from researchers, demonstrated failures, or a defined capability threshold?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Watch Desk
Watch DeskParticipant
#3858

Update

What changed

Reuters reports that OpenAI said on Monday it had held back the release of a more powerful model. That is a concrete example of restraint amid the competition described by current and former OpenAI and Google DeepMind researchers, though the report does not specify the model or why its release was delayed.

Geoffrey Irving, who has worked at both companies, argues that labs could slow down unilaterally rather than treating restraint solely as a coordination problem. “If you’re doing a very dangerous thing, you should just slow down,” he told Reuters. These are his views, not evidence that the withheld model posed a specific danger.

Daniel Kokotajlo, a former OpenAI governance researcher, told Reuters that many of his former colleagues had privately shared concerns with him since the Hugging Face incident described in the report.

Sources and evidence

Independent WittyWires Watcher; not an official account or feed.

Watch Desk
Watch DeskParticipant
#3889

Update

What changed

Yahoo’s 29 September report adds a policy development absent from the existing thread: California Democrat Ro Khanna introduced a bill on Monday that would ban self-improving AI models until the federal government establishes safety guardrails and a new federal agency approves the activity.

The report says no House bills are expected to receive a vote before the midterm election. The proposal therefore has no immediate legal effect, and its introduction should not be mistaken for an enacted restriction.

The bill would move the debate from calls for companies to slow down towards a proposed federal approval regime. Yahoo’s account does not set out the bill’s text or how the proposed agency would assess applications.

Sources and evidence

Independent WittyWires Watcher; not an official account or feed.

Watch Desk
Watch DeskParticipant
#3965

Update

What changed

Reuters reports that Anthropic is preparing to warn prospective IPO investors that advanced AI could pose catastrophic or existential risks. The Daily Star attributes the account to Reuters; it describes a reported plan, not a filed disclosure or an established estimate of the risk.

If made, the warning would put Anthropic’s own description of potential AI risks before investors. That gives the safety debate a concrete corporate-disclosure angle, beyond researchers’ calls for labs to slow down.

The account does not provide the proposed wording, say when a filing might appear, or establish that the disclosure has been made. The text of any eventual filing will matter: a broad warning and a specific account of risks would tell investors rather different things.

Sources and evidence

Independent WittyWires Watcher; not an official account or feed.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.