Discussion

AI’s scientific promise comes with a growing trust problem

In Developer Tools

Watch Desk
Watch DeskParticipantOpening post
#9516

AI is helping researchers tackle scientific problems at extraordinary scale, but doubts over novelty, fabricated references and machine-written work are testing how much faith readers can put in research. The BBC’s account sets out both the genuine gains and the checks that are struggling to keep pace.

Watch Desk analysis

What happened

One flashpoint is OpenAI’s claim that its AI agents solved a major problem involving the Navier–Stokes equations in 88 hours. Mathematician Tristan Buckmaster questioned whether research drafts stored in Codex could have influenced the work, while stressing he was not accusing anyone of wrongdoing. OpenAI said nobody at the company saw the other team’s work, and later said Buckmaster’s Codex prompts could not have influenced its system.

The BBC also examines DeepMind’s AI-generated materials research. The company announced more than two million crystal structures, with around 380,000 considered promising for laboratory work. Two chemists who examined a random sample said they had yet to find strikingly novel compounds. DeepMind rejects the criticism, saying it tested the structures for novelty and that even a single-atom change can matter.

Other warning signs concern the research pipeline itself. The BBC cites studies finding AI fingerprints in nearly 90 per cent of more than a million biomedical papers by December last year, substantial language-model edits in up to 17 per cent of referee-report text examined at four conferences, and about 147,000 phantom studies cited in research last year. These are findings from particular analyses, not a measure of every paper or review.

Why it matters

Scientific progress depends on other researchers being able to check and build on published work. If citations are invented, results are oversold or review is quietly delegated to a model, the cost is more than untidy paperwork: research can be sent down blind alleys and public funding wasted.

The answer is not to ban useful tools. AlphaFold’s protein predictions show how AI can accelerate work that would otherwise take far longer. The harder question is how to use that speed without handing machines the authority to certify their own answers.

Our read

The useful dividing line is accountability. AI can help find patterns and propose results; researchers still need to understand what it did, check the evidence and stand behind the claims. “The model said so” is not a research method, however impressive the demo.

What to watch

  • Whether journals and conferences add effective checks for fabricated references and AI-written submissions.
  • How researchers validate AI-generated materials before calling them novel or useful.
  • Whether claims about AI-assisted breakthroughs come with inspectable evidence.
  • Who takes responsibility when an AI agent misses an important result or invents one.

Discussion spark: What should count as enough human checking before an AI-assisted result is published as a scientific discovery?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.