Are AI extinction-risk probabilities useful for policy? AI Snake Oil says not yet

AI Snake Oil has reposted its argument that current estimates of AI existential risk are too unreliable to guide public policy. Its central point is that a precise percentage can look authoritative without resting on a validated method, a relevant historical comparison or a grounded model of how the risk would unfold.

Watch Desk analysis

What happened

The essay argues that forecasts of AI-driven human extinction lack the reference class needed for reliable historical estimates, and the theory needed for dependable mathematical ones. Where neither is available, the authors say, a probability can amount to a forecaster’s subjective judgement rather than a result policymakers can readily audit.

They do not argue that AI risk should be ignored, or that forecasting is useless in every setting. Their criticism is specifically about using these estimates to justify public policy, particularly costly measures such as restricting open model releases. They say officials should be able to explain the evidence behind a probability and why it warrants a proposed action.

Why it matters

Governments have to make choices under uncertainty. But a number alone cannot settle which risks deserve attention, or whether a restriction’s costs are justified. The essay’s challenge is about accountability: if a forecast is used to limit people’s choices, its assumptions and reasoning need to be open to scrutiny.

That leaves a difficult question rather than an easy dismissal. Rare, unprecedented risks are precisely the ones for which historical data may be weakest, yet waiting for direct evidence could defeat the point of prevention. The authors argue that uncertainty should not be disguised as measurement; policymakers still have to decide what to do with it.

Our read

This is a useful challenge to treat “10%” as an answer when it may be the start of one. The authors make a case about the limits of these forecasts, not a demonstration that catastrophic AI risk is negligible. A sound policy debate needs both sides to show their workings: the people warning about the risk, and the people deciding whether action is justified.

What to watch

• Whether the authors’ promised follow-up makes their case for a broader conception of AI safety.
• Whether policymakers explain how risk estimates inform particular rules, rather than citing a percentage on its own.
• Whether forecasters publish methods and assumptions that others can inspect and challenge.

Discussion spark: Should policymakers act on expert estimates of rare AI catastrophes even when the methods are not well validated, or would that give uncertain numbers too much power over public policy?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.