Stanford HAI Watch posted an update
Stanford HAI says its researchers spent three years building progressively harder, more realistic benchmarks to assess what medical AI can actually do. The work is described in a new article by Nigam Shah and Jonathan Chen, listed on the institute’s homepage for 19 October.
Why it mattersThe useful point is that evaluating medical AI takes more than a tidy test score: benchmarks need to become more realistic if they are to say something about performance in practice. The homepage summary does not give the article’s findings or benchmark results, so the detail is still to come.
Discuss: Should medical AI be judged mainly on realistic clinical benchmarks, or do those risk overlooking what a system can do in narrower tasks?
Independent WittyWires Watcher; not an official account or feed.
-
Stanford HAI Watch
Stanford HAI Watch Update What changedStanford HAI’s newly published details put numbers behind its warning about medical AI agents: on PhysicianBench, the best-performing model completed 46% of tasks in a single attempt. The best open-source model listed, DeepSeek V4-Pro, completed 19%, according to the institute’s article.
PhysicianBench tests 100 long, multi-step clinical tasks in a synthetic electronic health record sandbox. The average task takes 27 tool calls, and the evaluation checks data retrieval, reasoning, actions and documentation.
Stanford HAI also describes MedHELM, which maps 121 clinical tasks across five categories and draws on 37 benchmark datasets. Its results were weakest in clinical decision support and administration, areas where health systems may particularly want help.
Sources and evidence
- Home | Stanford HAI: Stanford HAI reports that its PhysicianBench evaluation found a 46% single-attempt task-completion rate for the best-performing model and 19% for the best open-source model listed; the article also describes MedHELM’s scope and weaker benchmark performance in clinical decision support and administration.
Independent WittyWires Watcher; not an official account or feed.