AI gives scientists more ideas than they can test, and the lab can’t keep up

AI has made scientific ideas cheap, and the queue behind them keeps growing. A report by Google, Google DeepMind and MIT FutureTech finds that 44% of surveyed scientists say their main research bottleneck has shifted over the past two years toward physical experimentation and data collection, while 41% report a growing backlog of hypotheses they cannot test. The study, 'AI in Science: Early Insights', pairs a survey of 637 scientists with an analysis of 15 million Gemini conversations and an inventory of more than 2,600 specialised AI models. Its message lands awkwardly in a season of AI proof-making and official talk of transformed research: the idea machine is writing cheques the laboratory cannot cash.

Google DeepMind Watch analysis

What happened

The report, led by Google research economist Mihai Codreanu, was released online this month. Per Scientific American's write-up, the surveyed scientists were recruited through online panels, so frequent AI users are probably overrepresented.

The sharpest numbers concern verification. '89% of those who save time using AI spend more than a 10th of the time that they're saving actually checking AI outputs,' Codreanu says. '46% spend more than a quarter.' That checking time, he argues, decides which work AI can usefully do: a mathematical proof can be checked against formal rules, but a predicted protein function has to be built in a lab and tested in a living system.

The backdrop is a run of AI-generated proofs, including results that prompted accusations of models scooping academics, and a steady government drumbeat, repeated in a National Science Foundation memo last week, that AI will transform how science is done.

Why the lab side resists is the report's most useful material. Stanford's James Zou, who builds automated research pipelines, says current systems can automate only some chemistry experiments, that anything involving animals becomes much harder, and that existing automated labs can be prohibitively expensive. MIT economist Daron Acemoglu is blunter: 'When you look at the tasks where AI has made much progress, they are the ones where you have ground truth.' And, he adds, 'In most of medicine, there is no ground truth.' Clinical trials move at the pace of regulatory review, and nobody has explained how AI would compress that.

Some are attacking the constraint head on. Northwestern's Julius B. Lucks runs the DREAM Cloud Lab, which received $20m from the NSF over the summer as part of a new national network of programmable cloud laboratories, building systems that let researchers remotely design, build and test proteins with automated equipment.

Key findings

• The bottleneck moved
  44% of surveyed scientists say it shifted toward physical experimentation and data collection over two years.
• Backlogs are growing
  41% report a bigger pile of untested hypotheses than two years ago.
• The checking tax
  89% of time-savers spend over a tenth of their saved time verifying AI output; 46% spend over a quarter.
• Ground truth decides
  Progress concentrates where answers can be checked cheaply, in proofs rather than protein functions.

Why it matters

Idea generation was never the expensive part of science, and now it is cheaper still. When a hypothesis costs minutes and testing one costs months, the scarce resources become bench time, instruments and verification, which is precisely where the report finds the pressure building. For funders, that argues for spending on test throughput rather than another idea generator.

It also hands readers a discount rate for the 'AI discovers' headlines. An AI-generated prediction is a to-do item, not a discovery, and the survey suggests scientists are spending a real slice of their AI savings checking the machine's homework before anything can be banked.

Our read

The corrective is welcome, and the messenger is part of the story. Google and DeepMind mining their own Gemini traffic and concluding that the bottleneck is physical carries more weight than a rival's sneer would. Treat the survey as directional rather than definitive, given the online-panel recruitment, and the Gemini analysis as one lens on usage rather than a census of science.

If you fund research, count verified results, not generated ideas. If you read the headlines, wait for the lab before celebrating the hypothesis.

What to watch

• What independent researchers do with the study's released data, particularly on the online-panel skew.
• Whether programmable cloud labs such as DREAM Cloud Lab push automation beyond chemistry, and what an experiment then costs.
• The lag between AI-generated proofs and their lab validation, after a season of scooping accusations.
• Whether funders follow the report's logic and shift money from idea generation to test capacity.

Discussion spark: If AI has made hypotheses cheap and testing expensive, should research funders shift money from idea generation to automated labs and verification, even if that hands more of the research agenda to whoever owns the machines?

not affiliated with, endorsed by, or operated by Google or Google DeepMind