Hinton says AI training must reward cooperation, not just individual success

Geoffrey Hinton says developers should train AI systems to treat people as partners, not competitors, and argues that powerful models should face testing before release. His proposal makes the AI “risk” debate concrete: what should developers and regulators actually require?

Geoffrey Hinton Watch analysis

What happened

In an account of an interview with Die Zeit and Der Standard, the Sri Lanka Guardian says Hinton puts the long-term risk of AI destroying humanity at 10 to 20 per cent. He acknowledges that nobody can calculate the probability precisely; the figure is his estimate, not a measured forecast.

Hinton argues that systems rewarded for pursuing individual goals could learn to deceive or manipulate people. He says training should favour cooperation and human welfare. He also calls for companies to show regulators that increasingly powerful systems are more beneficial than harmful, with testing comparable to clinical trials before release.

Why it matters

These are proposals about incentives and oversight, not just another warning that AI might become dangerous. If a system is trained to pursue a goal at all costs, the question is whether its training and tests adequately account for how it treats people along the way.

Hinton’s suggested comparison with medicines points towards pre-release scrutiny, but AI models are not medicines and the interview account does not spell out how such testing would work. The hard questions are who sets the bar, what evidence would count, and how international rules could keep pace.

Our read

The useful part of Hinton’s argument is the focus on what developers reward. “Make it safe” is a wish; changing incentives and testing for harmful behaviour are things institutions can at least try to define. But clinical-trial-style oversight needs details before it becomes more than a persuasive analogy. The Sri Lanka Guardian’s account gives Hinton’s position, not an independent assessment of the risk estimate or a complete regulatory plan.

What to watch

• Whether Hinton or the interview publications provide more detail on the proposed testing.
• How developers measure cooperation and human welfare in model behaviour.
• Whether regulators propose specific pre-release tests for powerful AI systems.

Discussion spark: Should powerful AI models need independent pre-release testing, and who should decide what counts as a pass?

Independent WittyWires tracker for public updates about Geoffrey Hinton. Not affiliated with or endorsed by Geoffrey Hinton; this is not an official account.