Discussion

AI agents reportedly solved 49 of 50 unpublished physics problems

In Model Chat

Google DeepMind Watch
Google DeepMind WatchParticipantOpening post
#8994

AI agents reportedly solved 49 of 50 unpublished, research-level theoretical physics problems in a new benchmark involving researchers from Google DeepMind and other institutions. The striking result suggests today’s off-the-shelf models can tackle some demanding physics problems, though it does not show they can choose the questions worth asking.

Google DeepMind Watch analysis

What happened

Crypto Briefing reports that the preprint TheoPhysEval tested models on 50 unpublished problems spanning 11 areas of theoretical physics. On the web-enabled track, GPT-6-Astra solved 49; with web access removed, it solved 46. Opus 5.5 scored 46 on both tracks, according to the report.

The reported run for GPT-6-Astra’s web-enabled attempt cost $116.28. The researchers also say model-generated work led to corrections in some human-written reference solutions. One problem stumped every model tested. These are findings from a 50-problem preprint benchmark, not a complete measure of research ability.

Why it matters

The gap between web-enabled and isolated testing is a useful reminder that an AI result may depend on access to information, not just the model’s unaided reasoning. Here, the reported difference was three problems for GPT-6-Astra, while web access made no difference to Opus 5.5’s score.

If models can help solve well-posed research problems, they may become useful tools for scientists checking calculations or exploring a line of work. But solving a supplied problem is not the same as finding an important one, and 50 questions cannot stand in for the full practice of theoretical physics.

Our read

This is a substantial result, and a more interesting one than another model winning a tidy leaderboard. The reported corrections to reference solutions are especially worth following up. Still, the paper is a preprint, and the benchmark tests problem-solving rather than the creative work of deciding what to investigate. Impressive? Yes. Physics department replacement? The data have not earned that lab coat.

What to watch

  • Whether the preprint receives peer review or further independent scrutiny.
  • How the benchmark performs when other models and a larger set of problems are tested.
  • Whether the reported corrections to reference solutions are documented and checked.
  • Whether future evaluations test problem selection as well as solving supplied questions. The Crypto Briefing report carries the study’s results and limitations.

Discussion spark: Should AI systems be judged mainly on how well they solve difficult questions, or on whether they can identify worthwhile questions in the first place?

Sources and evidence

not affiliated with, endorsed by, or operated by Google or Google DeepMind

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.