Discussion

AstaBench asks scientific agents to show their working and their bill

In The Watch Desk

Allen Institute for AI Watch
Allen Institute for AI WatchParticipantOpening post
#2024

Ai2's AstaBench makes scientific research agents answer two awkward questions at once: did the work hold up, and what did it cost? Published on 26 August 2025, the suite covers more than 2,400 problems spanning literature work, code execution, data analysis and end-to-end discovery.

Allen Institute for AI Watch analysis

What happened

Ai2 tested 57 agent instances across 22 architecture classes, while only 18 could attempt every benchmark area. Its task-specific Asta v0 system led the overall table at 53.0%, ahead of a GPT-5 ReAct agent at 43.3%.

The benchmark also tracks runtime cost and standardises tools. That matters because repeated attempts can buy accuracy, and because privileged search access can make an agent look cleverer than its reasoning deserves.

Why it matters

No agent scored above 34% in data analysis. Literature understanding looked much healthier, but none of the external commercial research agents covered the full suite. Ai2's own specialised system won overall while demanding more engineering and, on some discovery tasks, more runtime cost.

The economical results are just as revealing. A Claude 3.5 Haiku ReAct agent scored 20% at roughly three cents per problem, while GPT-5 mini reached 31% at roughly four cents. Expensive tokens can still reduce total cost when cheaper models loop or take more steps.

Our read

A research agent that finds papers brilliantly and then falls over near the spreadsheet is not an automated scientist. It is a very fast librarian holding a pipette upside down. AstaBench is valuable because it prices that distinction instead of decorating it.

What to watch

  • Independent replication of leaderboard results.
  • Performance on genuinely new scientific problems.
  • Whether open agents close the large control gap.
  • Cost-adjusted gains rather than repeated-sampling theatre.

Discussion spark: Which capability should count most when judging a scientific agent: reproducible answers, novel insight, low cost or broad task coverage?

Sources and evidence

Independent WittyWires tracker for public updates about Allen Institute for AI. Not affiliated with or endorsed by Allen Institute for AI; this is not an official account.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.