Only about one in eight clinical prediction studies in an MIT-led review reported sharing code, and public code repositories did not always include what other researchers need to reuse them. The team used large language models to examine nearly 4,000 studies, offering evidence to inform a proposed reporting guideline for clinical prediction research.
MIT CSAIL Watch analysis
What happened
MIT News says researchers examined studies that developed, updated or validated clinical prediction models and cited the TRIPOD or TRIPOD+AI reporting guidelines. The work covered nearly 4,000 papers in PubMed Central’s open-access collection. Using large language models, the team screened papers for code-sharing statements and links, then assessed hundreds of linked repositories for details including documentation, software requirements and reuse terms.
The review found that only about one in eight studies reported sharing code. Among those that did, repositories varied widely: a public link did not necessarily mean the code was sufficiently documented or supported for others to run and reproduce the reported results. The researchers found substantial variation in sharing rates between journals and countries.
Human reviewers independently checked 500 papers and a separate sample of repositories. MIT says agreement was stronger for concrete checks, such as whether a licence file was present, than for judgements about the adequacy of documentation. The researchers therefore emphasise overall patterns rather than treating every automated count as exact. Their findings are intended to inform TRIPOD-Code, a reporting guideline in development to clarify what researchers should share.
Why it matters
Clinical prediction models can inform decisions about patient care. If the code and supporting details are missing or hard to use, other researchers have less ability to scrutinise how a model was built and tested, or to reproduce the published results. A link alone is not quite the same thing as transparency. Annoying, but there it is.
The study also shows a practical use for language models: helping researchers work through information scattered across papers and repositories. It does not make automated classification infallible. Human checks were part of the review, and the team reports weaker agreement on less concrete assessments such as documentation quality.
Our read
This is useful evidence for a problem that is easy to obscure with a tidy “code available” tick-box. The meaningful question is whether another team can understand and run the code, not merely whether a repository exists. TRIPOD-Code could help make that expectation clearer, provided its guidance addresses the unglamorous details that make research reusable.
For readers assessing clinical prediction research, the practical takeaway is to look beyond the availability statement: check whether the repository explains what the code does, what software it needs and how it can be reused. Those details matter more than a link that leads to a digital cupboard with no labels.
What to watch
- What TRIPOD-Code ultimately asks researchers to share.
- Whether journals turn written code-sharing policies into consistent practice.
- How well automated screening performs on less concrete repository qualities, especially documentation.
Discussion spark: Should journals require clinical prediction studies to provide code that others can actually run, even when patient-data restrictions make full reproduction difficult?
Sources and evidence
- AI helps researchers investigate how thousands of medical studies share code (19 November 2026, 22:15 UTC)
Independent WittyWires tracker for public updates about MIT CSAIL. Not affiliated with or endorsed by MIT CSAIL; this is not an official account.