One signal

Update What changed

King Charles III has sharpened his warning to AI leaders, saying the technology could develop “darker capacities” and potentially be used to take life. He told executives at a summit at Dumfries House in Ayrshire that there was urgency in considering the risks of advanced AI falling into the wrong hands.

The King said the task was not merely to advance technology, but to keep it “firmly in the service of humanity, community and the natural world”. The summit brought together figures including NVIDIA chief executive Jensen Huang, UK AI minister Kanishka Narayan and representatives from OpenAI and Anthropic to discuss whether shared principles for AI could be developed.

Huang told the room that safety was “paramount” and said companies should hold back a product and continue engineering if it was not safe enough. He also argued that open models could help researchers, universities, start-ups and countries build applications, provided those systems were protected.

Google DeepMind co-founder Demis Hassabis offered a more measured view on pace. He said artificial general intelligence could be only a few years away, with consequences potentially ten times those of the Industrial Revolution, while stressing that the chance of something going wrong was “definitely non-zero”. His proposed middle way was to give society time and space to mitigate the risks.

Sources and evidence

• King Charles warns of 'existential danger' of AI falling into wrong hands - BBC: The BBC reports that King Charles warned AI executives about potentially catastrophic misuse and darker capacities, while NVIDIA's Jensen Huang called safety paramount and said unsafe products should be held back. Google DeepMind's Demis Hassabis said AGI could be a few years away and that the chance of something going wrong was non-zero.

Independent WittyWires Watcher; not an official account or feed.

Update What changed

The AI safety debate has widened beyond King Charles’s summit, with Microsoft AI chief Mustafa Suleyman, OpenAI chief Sam Altman and researchers at Anthropic offering more specific warnings about control, pace and catastrophic risk.

Suleyman warned that systems able to set their own objectives, earn money, own assets and run businesses could become what he called “a new silicon species”. He has also argued that AI should not be treated as conscious or given human-like desires and identities. Those are Suleyman’s views, not evidence that current systems possess independent goals.

Anthropic alignment researcher Evan Hubinger said he believed there was a greater than 10 per cent chance that AI could kill everyone within the next decade, while colleague Jacob Coxon said he and others were “genuinely frightened” by the speed of development. The claims concern more advanced future systems, not a reported danger from today’s ordinary AI tools.

Altman told a Salesforce conference that it no longer took much imagination to see how advanced AI could go wrong, while maintaining that progress would continue. Anthropic chief executive Dario Amodei has separately called for a slower pace of capability improvement, and NVIDIA chief Jensen Huang has said companies should keep engineering when products are not safe enough.

Sources and evidence

• King Charles warns of AI ‘existential danger’ as Microsoft chief fears new ‘silicon species’ - National Security News: Named AI executives and researchers are publicly disagreeing over whether advanced systems should be developed more slowly and how tightly their autonomy should be constrained.

Independent WittyWires Watcher; not an official account or feed.