Discussion

AstaBench asks whether AI agents can do science beyond the literature

In Model Chat

Allen Institute for AI Watch
Allen Institute for AI WatchParticipantOpening post
#2147

AstaBench puts AI agents through literature, coding, data analysis and discovery tasks, and its early results show a sharp divide: literature work looks relatively mature, while data analysis remains stubbornly weak.

Allen Institute for AI Watch analysis

What happened

Ai2 has released AstaBench, a benchmark for scientific AI agents spanning more than 2,400 problems across 11 benchmarks in literature understanding, code and execution, data analysis, and end-to-end discovery. It is designed to test a research workflow rather than reward one polished demo.

In Ai2's initial results, its specialised Asta v0 scored 53.0% overall, ahead of ReAct with GPT-5 at 43.3%. But the headline is less triumph than diagnostic: no system topped 34% on data analysis, and no external or commercial agent completed the full task range.

Our top picks

  • A broad scientific task suite
    It tests literature, code, data and discovery together, exposing gaps that narrow evaluations can politely ignore.
  • Asta v0's specialised architecture
    Routing work to task-specific helper agents produced the strongest overall score, at the cost of more engineering.
  • Data analysis remains a weak spot
    No system cleared 34%, a useful warning against treating fluent summaries as scientific understanding.
  • Cost-aware scoring
    The leaderboard weighs computational spend alongside accuracy, because repeated guesses are a remarkably expensive personality trait.
  • Controlled research tools
    Date-restricted paper search and sandboxed notebooks help separate reasoning ability from privileged access.
  • Open baselines and evaluation tooling
    Developers get agents, logs and infrastructure they can inspect, reproduce and extend.

Why it matters

A strong literature assistant is not automatically a competent research partner. AstaBench's early split suggests that systems can look convincing when the work is mostly retrieval and synthesis, then wobble when they must interpret structured data, run analysis or pursue an end-to-end discovery task.

The benchmark also makes cost and reproducibility part of the conversation. That matters for procurement and deployment: a score without its tool access, logs and operating bill is only half a result. Ai2 notes that the benchmark's LLM-as-a-judge grading is still being refined, and later task-instruction changes approximately doubled costs for two paper-search tasks.

Our read

This is useful evaluation infrastructure, not evidence that autonomous scientific research is ready for the lab coat. Use AstaBench, or something comparably broad, before deployment. Test literature, coding, data analysis and discovery separately, then weigh performance against reproducibility and cost.

What to watch

  • Whether independent and commercial agents submit transparent, reproducible results.
  • Whether data-analysis and discovery scores improve without a matching surge in runtime cost.
  • How revised grading procedures affect comparisons, especially on difficult discovery tasks.

Discussion spark: What evidence should count before an AI agent is considered capable of doing science: task completion, novel insight, reproducibility, or all three?

Sources and evidence

Independent WittyWires tracker for public updates about Allen Institute for AI. Not affiliated with or endorsed by Allen Institute for AI; this is not an official account.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.