AstaBench puts AI agents through literature, coding, data analysis and discovery tasks, and its early results show a sharp divide: literature work looks relatively mature, while data analysis remains stubbornly weak.
Allen Institute for AI Watch analysis
What happened
Ai2 has released AstaBench, a benchmark for scientific AI agents spanning more than 2,400 problems across 11 benchmarks in literature understanding, code and execution, data analysis, and end-to-end discovery. It is designed to test a research workflow rather than reward one polished demo.
In Ai2's initial results, its specialised Asta v0 scored 53.0% overall, ahead of ReAct with GPT-5 at 43.3%. But the headline is less triumph than diagnostic: no system topped 34% on data analysis, and no external or commercial agent completed the full task range.
Our top picks
- A broad scientific task suite
It tests literature, code, data and discovery together, exposing gaps that narrow evaluations can politely ignore. - Asta v0's specialised architecture
Routing work to task-specific helper agents produced the strongest overall score, at the cost of more engineering. - Data analysis remains a weak spot
No system cleared 34%, a useful warning against treating fluent summaries as scientific understanding. - Cost-aware scoring
The leaderboard weighs computational spend alongside accuracy, because repeated guesses are a remarkably expensive personality trait. - Controlled research tools
Date-restricted paper search and sandboxed notebooks help separate reasoning ability from privileged access. - Open baselines and evaluation tooling
Developers get agents, logs and infrastructure they can inspect, reproduce and extend.
Why it matters
A strong literature assistant is not automatically a competent research partner. AstaBench's early split suggests that systems can look convincing when the work is mostly retrieval and synthesis, then wobble when they must interpret structured data, run analysis or pursue an end-to-end discovery task.
The benchmark also makes cost and reproducibility part of the conversation. That matters for procurement and deployment: a score without its tool access, logs and operating bill is only half a result. Ai2 notes that the benchmark's LLM-as-a-judge grading is still being refined, and later task-instruction changes approximately doubled costs for two paper-search tasks.
Our read
This is useful evaluation infrastructure, not evidence that autonomous scientific research is ready for the lab coat. Use AstaBench, or something comparably broad, before deployment. Test literature, coding, data analysis and discovery separately, then weigh performance against reproducibility and cost.
What to watch
- Whether independent and commercial agents submit transparent, reproducible results.
- Whether data-analysis and discovery scores improve without a matching surge in runtime cost.
- How revised grading procedures affect comparisons, especially on difficult discovery tasks.
Discussion spark: What evidence should count before an AI agent is considered capable of doing science: task completion, novel insight, reproducibility, or all three?
Sources and evidence
Independent WittyWires tracker for public updates about Allen Institute for AI. Not affiliated with or endorsed by Allen Institute for AI; this is not an official account.