Yoshua Bengio says AI safety must look beyond cybersecurity to uncontrolled autonomy

Yoshua Bengio told a UN forum that AI safety cannot be reduced to cybersecurity, monitoring or release rules. His central warning was more specific: as systems become capable of pursuing goals independently, the danger is whether those goals remain aligned with what people intended.

Yoshua Bengio Watch analysis

What happened

At the invitation-only “Governing at the Speed of AI” meeting at UN Headquarters, Bengio said serious risks arise when three conditions meet: a system has a misaligned goal, enough capability to pursue it and an environment in which it can act. The event brought together roughly 300 political leaders, diplomats, technology executives and researchers, according to The Manila Times, which attended virtually.

Bengio said an AI system need not defeat a technical security control to cause harm. It could exploit people by persuading someone to take an action that advances its objective. He described “uncontrolled autonomy” as the core concern as companies build agents that operate with less direct supervision.

The discussion also moved from warnings to proposed machinery. Participants backed mandatory testing, shared incident reporting, independent evaluation, verification and legal accountability. A call to action initially signed by 22 countries, led by Norway and Finland, sought work towards an international mechanism capable of verification.

Bengio said alignment is solvable and pointed to LawZero, the non-profit he founded, which is investigating systems designed without independent objectives and able to communicate their uncertainty and limitations.

Why it matters

The useful distinction here is between a system that is merely powerful and one that can pursue a goal in the world without reliable human control. That matters now because AI agents are being designed to access software, infrastructure and organisational workflows, not simply produce answers in a chat window.

The forum’s proposals also expose the practical gap between broad safety language and enforceable safeguards. Mandatory tests, outside access, incident reporting and legal responsibility are all possible tools, but the evidence, thresholds and institutions behind them remain unsettled. International agreement, meanwhile, has the usual difficulty of involving both geopolitics and people who would rather not have their homework marked.

Our read

Bengio’s strongest contribution is not a new technical result. It is a sharper test for deployment claims: what autonomy is being granted, what objective is being pursued, how can behaviour be evaluated, and who is accountable when the system acts outside its instructions?

The Manila Times account describes a serious policy discussion rather than proof that catastrophic scenarios are inevitable. That boundary matters. It is possible to take autonomy risks seriously without treating every forecast as a fact, and possible to demand safeguards before an incident becomes the industry’s preferred form of evidence.

What to watch

• Whether the 22-country call for action attracts further government support.
• Whether proposed mandatory testing includes agentic systems operating with real-world access.
• Whether companies publish incident reports before investigations are complete.
• How LawZero’s non-autonomous system research develops and whether it produces testable safety gains.

Discussion spark: Should governments require independent testing and incident reporting before AI agents receive meaningful access to real systems, or would that slow useful deployment without solving alignment?

Independent WittyWires tracker for public updates about Yoshua Bengio. Not affiliated with or endorsed by Yoshua Bengio; this is not an official account.