Astro2Method finds LLMs struggle to choose the right method for astronomy

A new benchmark finds that large language models can often identify suitable machine-learning methods in straightforward astronomy questions, but struggle when the evidence is incomplete. Adding retrieval or agent-style reasoning did not consistently solve the harder cases, according to the Astro2Method paper.

Watch Desk analysis

What happened

Astro2Method tests five LLMs on 500 expert-reviewed questions drawn from 346 astronomy papers. The questions cover choices between methods where the papers report clear performance differences, with explanations that are either complete or partial.

The paper reports 87.50% to 92.55% accuracy on the easier questions, but only 58.87% to 64.52% on the harder ones. Retrieval-augmented generation and most tested agent approaches, including ReAct, Self-Ask and CRITIC, did not bring consistent improvement using historical papers.

Web-searching agents improved mainly when they found the specific papers behind the questions. Giving models a source paper’s pipeline description, with method names masked, improved performance on the harder questions to match or exceed web search. The benchmark and findings are described in the Astro2Method paper.

Why it matters

Choosing a machine-learning method is not just a matter of knowing the names in the toolkit. The relevant experimental details can decide which method fits a particular problem, and the results suggest models struggle to infer those details from related studies and general domain knowledge alone.

For researchers considering LLMs as assistants, the practical lesson is to give them relevant source material and check their recommendations, especially when the question is difficult. A confident answer is not the same thing as a well-supported method choice.

Our read

This is a useful benchmark because it tests a real research task rather than asking models to recite AI trivia. Its most telling result is also refreshingly unmagical: access to the right experimental description mattered more than piling on extra reasoning or retrieval techniques. The paper’s findings concern this benchmark, not every astronomy task, but they make a strong case for judging research assistants on whether they can use the evidence in front of them.

What to watch

• Whether later benchmarks test method selection on new papers and research fields.
• Whether models can reliably identify and use the experimental details that matter.
• How well these benchmark results carry over to researchers’ real workflows.

Discussion spark: For AI research assistants, should developers prioritise better access to relevant papers, or better reasoning over the material models already have?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.