UniPat AI benchmark finds agents struggle to predict real lab results

AI agents tested by UniPat AI scored below 0.5 on average when asked to predict real wet-lab results, a sign that fluent scientific reasoning is not yet the same as anticipating what an experiment will find. The new benchmark covers 116 tasks drawn from 4,207 experiments across chemistry, materials science, biology and medicine.

Watch Desk analysis

What happened

UniPat AI says its Science Prediction Universe benchmark asks agents to rank the likely outcomes of real experiments before they see the results. The first round tested 13 configurations of 12 frontier models. Across configurations, only 34 to 52 per cent of tasks beat the benchmark’s chance threshold, according to the company’s results, reported by PanDaily.

The company says Claude Opus 5.5 recorded the highest mean score, 0.486. Most of its runs ended in safety refusals, however, and it was scored on fewer tasks. UniPat’s analysis also found that agents agreed with one another more often than they agreed with experimental results: 0.52 versus 0.33.

Why it matters

The test targets a useful gap in AI evaluation. An agent can search the web, use scientific software and produce a confident ranking; the harder question is whether that ranking matches what happens in the lab. Here, consensus among agents was no guarantee of agreement with experimental reality.

The public benchmark includes 100 tasks based on published papers. Another 16 tasks come from unpublished experiments with partner laboratories and are not public, according to UniPat. That mix could help researchers test performance on both documented results and less familiar experimental data, though the private tasks make some parts of the evaluation harder for outsiders to inspect.

Our read

This is a promising reality check, not a final league table for scientific AI. The results are UniPat’s own evaluation, and its top score needs context: Opus 5.5 was assessed on fewer tasks because many runs were refused. Still, asking models to predict measured outcomes is a more demanding test than asking them to explain a paper after the answer is known. Science has enough hindsight already.

What to watch

• Whether independent teams reproduce the results across the public tasks.
• How models perform as the benchmark adds experiments and new model configurations.
• Whether the private tasks remain available for independent scrutiny, or are replaced by a clear public evaluation protocol.

Discussion spark: Should scientific AI be judged mainly on predicting experimental results, or do benchmarks like this risk overvaluing one kind of scientific reasoning?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.