AI explanations can look convincing even when a model has learned nothing

Some AI explanations may look meaningful even when the model being explained has learned nothing, researchers warn. A study highlighted by Medical Daily says naive statistical tests flagged up to 100% of findings across six models when they were untrained, while comparing against a pool of matched untrained models brought false-positive rates close to expected levels.

Watch Desk analysis

What happened

The problem has an unlikely mascot: a dead salmon in an MRI scanner. In a 2005 demonstration, researchers found apparent brain activity in the fish when they tested thousands of voxels without correcting for multiple comparisons. Apply standard corrections, and the activity disappears.

Medical Daily says AI researchers see a related risk in interpretability: methods for explaining what a model has learned can produce plausible-looking results even for randomly initialised networks. A poster listed for a NeurIPS 2026 workshop reported false positives of up to 100% for naive significance tests across six models and three families of explanation methods. Comparing results with a pool of matched untrained models, ideally 20 or more, brought false-positive rates back near expected levels, the account says.

Why it matters

An explanation is often part of the case for trusting an AI system. If the same explanation could appear when the model has learned nothing, it is not much of a reassurance, especially when a system is being assessed for use in areas such as medicine.

The point is not that AI explanations are useless. It is that plausible is not the same as informative, and testing explanations against models that have learned nothing can help expose the difference.

Our read

The salmon is memorable; the statistical lesson is the keeper. Treat an attractive explanation as a claim to test, not a window into a model’s mind. That is less cinematic than a glowing neural network graphic, but considerably more useful.

What to watch

• Whether future interpretability studies compare their results with matched untrained models.
• Whether the workshop poster’s findings are independently tested and published in a full paper.
• Whether researchers and developers make these checks routine when using explanations to assess AI systems.

Discussion spark: Should evidence that an explanation beats a matched untrained model be a standard requirement before it is used to support trust in an AI system?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.