Goodfire’s interpretability thesis says neural networks may be more readable than they look

Goodfire researcher Nick Cammarata argues that many apparently tangled neural-network features may become understandable through further analysis by AI agents. The claim challenges a long-running assumption about how difficult it is to interpret what large models have learned, but researchers quoted in Latent Space’s roundup dispute how far the evidence goes.

Watch Desk analysis

What happened

Cammarata’s essay, “Neural Networks Are Already Clean”, offers a working theory that many known cases of polysemanticity and superposition, where model features appear entangled, can be resolved through agentic analysis. Latent Space says the argument draws on Cammarata’s earlier automated-interpretability work, which found only about 2% of features legible in 2023.

The roundup also records pushback. Neel Nanda says he still expects representational superposition and thinks weak activations may carry memorised facts. Leo Gao argues that explanations should be tested by how well they simulate neuron behaviour, while Jake Mendel flags the possibility that important activity sits outside the strongest activations. The essay itself is not yet a paper, the roundup notes.

Why it matters

If models’ internal features are easier to interpret than researchers assumed, that could make it more practical to investigate what they represent and how they behave. But “an agent produced an explanation” is not the same as “the explanation faithfully captures the model”. The disagreement is about what would count as convincing evidence, not merely whether AI can help with the analysis.

Our read

This is a worthwhile challenge to a familiar picture of neural networks as inscrutable knots. It is also a hypothesis under debate, not a settled result. The strongest next step would be explanations that survive independent tests, including checks that they predict what the relevant neurons actually do. Interpretability needs more than a tidy story; it needs a way to catch itself being wrong.

What to watch

• Whether Cammarata publishes the essay’s evidence and methods in a form others can test.
• Whether independent work reproduces the reported interpretability results.
• How proposed tests handle weak activations and possible superposition.

Discussion spark: What evidence should persuade researchers that an AI-generated explanation reflects a model’s real internal features, rather than just a plausible story?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.