🔍 Read the full analysis: The Fine Print In Microsoft’s AI-versus-Doctors Study on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A Science paper published October 22 reports that Microsoft’s MAI-DxO diagnostic system scored 83.2% on 304 complex case records, compared with 36.1% for generalist doctors using reference materials. The results come from a text-based simulation, not care delivered to real patients, and the study’s comparison does not include specialists or physician teams.
Microsoft’s MAI-DxO diagnostic AI scored higher than generalist physicians on a benchmark of complex medical cases, according to a peer-reviewed paper published October 22 in Science. The study reports 83.2% accuracy on 304 case records, versus 36.1% for doctors using textbooks and the internet, but the comparison took place in a text-based simulation rather than with real patients.
The paper, “Sequential diagnosis with language models,” describes the Sequential Diagnosis Benchmark, which begins with a short case summary. A doctor or AI system must request additional details, such as test results, from a simulated gatekeeper. The benchmark evaluates both the final diagnosis and the estimated cost of obtaining information. Microsoft’s system uses a language model in a panel-like process, with virtual agents proposing diagnoses, requesting tests and challenging one another.
Microsoft reported 83.2% accuracy for MAI-DxO on 304 complex cases drawn from the New England Journal of Medicine. The comparison figures were 36.1% for generalist physicians with textbooks and internet access and 22.3% without those resources. On a separate set of 325 emergency-department cases, the paper reports scores of 80.3% for the AI and 46.8% for physicians. These are results reported by the paper’s Microsoft-affiliated authors, not an independent replication.
The comparison involved generalists working alone, not specialists or clinical teams. The cases were historical and text-only; the exercise did not involve physical examinations, imaging in a real clinical workflow or treatment decisions for actual patients. The source account says the study also did not include healthy patients or harmless explanations among the complex NEJM cases, limiting what it can show about false alarms. A prospective study at Beth Israel Deaconess Medical Center is planned, with consenting patients interacting with the system alongside their usual care.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
What the Accuracy Gap Can Show
The findings matter because they offer evidence about a specific task: gathering information and proposing diagnoses from difficult, written cases. Unlike a conventional multiple-choice test that supplies all the facts, this benchmark asks the system to decide what to request next. That makes it a more involved test of diagnostic reasoning than simply selecting an answer from a complete case description.
But the reported gap does not establish that MAI-DxO is safer or more effective than doctors in routine care. The benchmark’s selected cases and solo-generalist comparison do not represent the full range of patients, uncertainty and collaboration found in hospitals. The absence of healthy patients and benign explanations also means the reported accuracy cannot answer how often the system would raise an unnecessary alarm in a less selected population.
For hospitals and patients, the practical distinction is between benchmark performance and clinical benefit. The paper reports model scores in a simulation; it does not report improved patient outcomes, shorter time to diagnosis or reduced costs in clinical practice. Those questions require testing with patients and comparison against the care teams who would actually use such a tool.
AI diagnostic tools for healthcare
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How the Benchmark Was Built
The paper follows a June 2025 preprint, which used an earlier version of the system running on OpenAI’s o3 model. The October paper is the peer-reviewed version, published in Science on October 22, 2026. All 15 authors are listed with Microsoft affiliations, according to the supplied source material. Peer review means the paper was assessed for publication; it does not mean independent researchers have reproduced its results.
The source account says important methodological details are drawn from the preprint because the full Science article is behind a paywall, and details may differ in the final paper. In that version, a model judged diagnostic answers against the published diagnosis, with physician re-grading showing good but imperfect agreement. When a requested test was missing from a case record, a gatekeeper model generated a synthetic result. Test costs were estimates based on billing codes and a 2023 U.S. hospital price list, plus a fixed $300 physician-visit charge; the authors described those estimates as first-order approximations.
The source also highlights the model-only result: GPT-5 scored 78.9% on the hard cases without MAI-DxO’s orchestration layer. The system’s additional layer therefore accounted for a smaller increase in reported accuracy than the overall difference between the AI and the physician comparison group might suggest. The source gives an estimated test cost of $8,409 per case for GPT-5 in that comparison.
“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”
— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch
As an affiliate, we earn on qualifying purchases.
What Patient Testing Must Establish
The benchmark does not establish how MAI-DxO performs with real patients, incomplete or conflicting histories, examination findings, imaging, or the workflow of a clinical team. It also does not show whether doctors would reach better results when using the system as an assistant, rather than being compared with it as solo participants.
The supplied account does not provide results from an independent replication or the planned prospective study. It is also unclear from that account how closely the final Science paper’s methods and figures match the preprint on which some methodological details are based. The cost estimates and the possibility of false alarms remain further limits on interpreting the headline accuracy figures.
As an affiliate, we earn on qualifying purchases.
The Planned Hospital Study
The next reported milestone is a prospective study at Beth Israel Deaconess Medical Center in Boston. Consenting patients are expected to interact with MAI-DxO alongside their normal care. Unlike the historical-case benchmark, that setting could test how the system works during actual patient encounters, although the supplied material does not give a start date, study size or outcome measures.
Results from that work will be needed to judge whether the benchmark advantage carries into clinical practice. Until then, the published figures support a narrower conclusion: MAI-DxO performed strongly on selected diagnostic simulations, while its effect on patient care remains unestablished.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did Microsoft’s MAI-DxO study report?
The paper reported 83.2% accuracy for MAI-DxO on 304 complex NEJM case records. It reported 36.1% for generalist doctors using reference materials and 22.3% without them.
Was the system tested on real patients?
No. The reported benchmark used historical, text-based cases in a simulation. A separate prospective study with consenting patients at Beth Israel Deaconess Medical Center is planned, according to the source material.
Were the doctors in the comparison medical specialists?
The comparison described in the source involved primary-care doctors and hospital generalists working alone. It did not compare MAI-DxO with specialists or physician teams.
Does peer review mean the results were independently confirmed?
No. The paper appeared in Science after peer review, but the supplied source says independent replication has not been reported. Peer review does not by itself establish that the results generalize to clinical care.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
