Duplicated eye scans may have inflated glaucoma AI results

Researchers found extensive duplication across three public datasets used to develop and test AI for glaucoma diagnosis, potentially making models look more accurate and equitable than they are. In a study published in JAMA Ophthalmology, they report that 1,031 of 1,302 matched scan pairs were identical, and that duplication inflated one model’s measured performance by 0.12 AUC points.

Watch Desk analysis

What happened

The researchers compared three datasets from a single laboratory: Harvard-GDP, Harvard-GF and Harvard-FairVision. Of 1,302 candidate pairs between Harvard-GF and Harvard-FairVision, 1,031 contained the same three-dimensional OCT scan. The researchers say this represents at least 31% of the Harvard-GF dataset.

In one test, a glaucoma classifier trained on Harvard-GF and evaluated against a Harvard-FairVision split containing many of the same scans scored an AUC of 0.90. On out-of-sample data, its score was 0.78. The researchers attribute the gap to data leakage: a test set is not a meaningful independent check if it includes images the model has effectively already encountered. Read the study summary.

Why it matters

Medical AI models are judged partly on whether they work on data beyond their training set. If supposedly separate datasets share patients or scans, benchmark results can flatter a model, and comparisons between models can become less useful. The researchers also warn that the overlap can make a less equitable model appear equitable.

The finding concerns these public glaucoma datasets; the report says the researchers could not establish how common the problem is across other public datasets. It is a specific warning about validation, not evidence that every medical AI benchmark is compromised.

Our read

This is the sort of unglamorous data problem that can make a polished performance figure wobble. For developers and evaluators, dataset independence deserves the same scrutiny as the model itself. The researchers recommend disclosing shared source cohorts and providing a privacy-preserving way to verify that datasets are independent.

What to watch

• Whether the dataset developers disclose shared cohorts and offer a way to verify independence.
• Whether other medical AI researchers audit public datasets for overlap.
• Whether future benchmarks report performance on genuinely independent data.

Discussion spark: Should medical AI dataset creators be expected to prove that their test data are independent, or is that an unreasonable burden when patient privacy is at stake?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.