Researchers compare performance of large language models with human physicians in real-world medical settings
A recent study published in the journal Science has shed light on the performance of large language models in medical contexts, including real-world emergency room cases. Researchers from Harvard Medical School and Beth Israel Deaconess Medical Center tested OpenAI’s o1 model against human physicians in a series of…
The study focused on 76 patients who visited the emergency room at Beth Israel, comparing the diagnoses generated by the o1 model with those made by two internal medicine attending physicians. The AI-generated diagnoses were then assessed by a third pair of physicians who did not know which came from humans and whic…
According to the study, the o1 model performed remarkably well in these experiments, achieving accuracy comparable to that of human physicians in 67% of triage cases. In contrast, one physician achieved an exact or close diagnosis only 55% of the time, while the other physician did so just 50% of the time.
While the study’s findings are promising, it is essential to note that they do not suggest AI is ready to make life-or-death decisions in emergency rooms. Instead, the researchers emphasize the need for further trials to evaluate AI’s potential in these settings.
What matters
- Study finds OpenAI’s o1 model achieves accuracy comparable to two human doctors in 67% of triage cases
- Experiment involves 76 patients and compares AI-generated diagnoses to those made by human physicians
- Researchers emphasize need for further trials to evaluate AI’s potential in emergency room settings
Why it matters
Researchers emphasize need for further trials to evaluate AI’s potential in emergency room settings
This GenAI News article was prepared in original wording using reporting and materials published by TechCrunch AI. Source reference: https://techcrunch.com/2026/05/03/in-harvard-study-ai-offered-more-accurate-diagnoses-than-emergency-room-doctors/.
Drafted by the GenAI News review pipeline.