AI models scored as low as 30 per cent on tests of difficult physics problems, but reached as high as 94 per cent after researchers corrected errors and ambiguous definitions, according to New Scientist. The result raises a useful question: are models struggling with physics, or with tests that fail to give them enough information?
Watch Desk analysis
What happened
John Sous of Yale University told New Scientist that his team found problems in the benchmarks, including errors and unclear definitions. After correcting them, scores rose to as high as 94 per cent. Those results are Sous’s account of the corrected tests, not proof that AI can do every kind of physics well.
The article also describes work in Logan Wright’s Yale laboratory, where researchers use Anthropic’s Claude to control laser pulses and collect data on their effects on rubidium atoms. Wright says experiments can run overnight and complete as much work as a person would in a week. The work is constrained: the experiment’s observations and actions must be made accessible through code.
Read the New Scientist report.
Why it matters
A benchmark is only as informative as the problem it asks. If a question is badly specified, a low score may say as much about the test as the model. Correcting the physics questions produced a striking change in the reported results, though it does not tell us how models perform across the whole field.
There is a more practical development alongside the scorecard debate: AI is being used to help run real, tightly controlled experiments. That can speed up exploration when instruments, inputs and outputs are code-accessible. It is a long way from handing a model the keys to all of theoretical physics, which is not currently how laboratories work.
Our read
The strongest point here is not that AI has suddenly solved physics. It is that weak tests can badly misdescribe what a system can do, while carefully designed experiments offer a more concrete way to find out. Treat the 94 per cent figure as a result from corrected questions, not a universal physics grade.
For researchers, the sensible next step is to report what was changed in a benchmark and test AI on specific experimental tasks, with scientists able to inspect the methods and results. The interesting frontier is not just whether a model can answer a question, but whether it can help produce evidence about the world.
What to watch
- Whether Sous and colleagues publish the corrected questions and fuller benchmark results.
- How Claude performs across more laser-pulse experiments, and how researchers assess its suggestions.
- Whether AI can contribute to less tightly controlled physics problems, where the right question is not already known.
Discussion spark: When an AI physics benchmark gives a low score, should the default assumption be that the model failed, or that the test may have failed to ask a well-defined question?
Sources and evidence
- Source update (16 October 2026, 06:00 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.