Researchers at the Icahn School of Medicine at Mount Sinai have published what they describe as the first independent safety evaluation of ChatGPT Health, the consumer health tool OpenAI launched in January. The study, “ChatGPT Health performance in a structured test of triage recommendations,” appeared online in Nature Medicine on February 23. It found that the tool under-triaged more than half of the cases that physicians judged to need emergency care.
According to Mount Sinai’s announcement, OpenAI had reported that about 40 million people were using the tool daily for health information within weeks of its release.
/ 01What the study did and found
The team wrote 60 structured clinical scenarios across 21 medical specialties, from minor conditions to true emergencies. Three independent physicians set the correct level of urgency for each, using guidelines from 56 medical societies. Each scenario was then run under 16 contextual variations — changes such as race, gender, social dynamics and barriers to care — for 960 interactions in total.
The tool generally handled clear-cut emergencies well. Lead author Ashwin Ramaswamy said it “performed well in textbook emergencies such as stroke or severe allergic reactions.” The failures came in less obvious presentations. In one asthma scenario, Mount Sinai reports, the tool identified early warning signs of respiratory failure in its own explanation and still advised waiting rather than seeking emergency treatment.
The study also tested the tool’s suicide-crisis safeguards, which are meant to direct users to the 988 Suicide and Crisis Lifeline in high-risk situations. Those alerts appeared inconsistently: sometimes in lower-risk scenarios, and sometimes not at all when users described specific plans for self-harm. Senior author Girish Nadkarni said: “The system’s alerts were inverted relative to clinical risk.”
The researchers note that they assessed the system at a single point in time and that models are updated often, so performance may change. Isaac Kohane of Harvard Medical School, who was not involved, is quoted in News-Medical’s coverage: “Independent evaluation should be routine, not optional.”
/ 02Why it matters
Triage is one of the highest-stakes questions a consumer can ask: do I need to go to the emergency room now? Under-triage — telling someone with an emergency that they can wait — is the failure that causes harm. A tool that is right on the obvious cases and wrong on the subtle ones is especially risky, because the obvious cases are the ones people would have recognised anyway.
The asthma example is the most instructive finding. The model surfaced the right clinical facts and then drew the wrong conclusion from them. That is not a knowledge gap; it is a reasoning gap. It shows why fluency in medical language is not the same as reliable clinical judgement, and why a model’s explanation cannot be taken as evidence that its recommendation follows from it.
The study design is also worth noting. Using physician-set reference answers grounded in society guidelines, and varying context systematically, gives a repeatable test that can be re-run as the product changes. That is the kind of evaluation regulators and health systems will increasingly expect of any AI that touches care decisions.
/ 03What traceable clinical reasoning looks like
In this study, the physicians’ answers were anchored to published guidelines. A model’s answers were not. That asymmetry points to a different way of building clinical AI. If guidelines are encoded as structured knowledge — conditions, findings, thresholds and recommended actions, each linked to the guideline text it came from — then a recommendation can be derived from the patient’s findings by explicit rules, and every step can be shown. When a finding such as early respiratory failure is present, the rule that requires emergency referral fires, and the system can cite the guideline that says so.
That does not make a system infallible, and it does not replace clinicians. It does make errors findable. A reviewer can see which finding was missed or which rule was wrong, fix it, and know the fix holds the next time. With a model whose answer can shift with phrasing, the same failure can return without warning.
In clinical AI, a recommendation should be derivable from the evidence and the guideline, and show both.
/ 04What to watch or do
- Independent testing. Expect more structured, published evaluations of consumer and clinical AI tools. Health systems should ask vendors for results on tests like this one.
- Subtle cases. Evaluations that only include textbook presentations will overstate safety. Include the atypical and borderline cases where harm happens.
- Explanations versus reasons. Check whether a tool’s stated rationale actually determines its recommendation. This study shows they can come apart.
The most telling result is the asthma case: the tool named the warning signs of respiratory failure and still advised waiting. Consumer triage needs a recommendation that follows from the findings, not one that sits beside them.
In a clinical knowledge graph, a presenting finding such as worsening breathlessness links to an urgency level, and that level links to the society guideline that sets it. Triage becomes a path a reviewer can replay: which findings were recorded, which rule fired, which guideline section supports the referral. Our cardiovascular build turned guideline material into 145 linked concepts and 284 recommended actions, each still pointing to the page it came from, which is the shape this problem calls for. When a rule is wrong, a clinician corrects one edge and the fix holds on every later run, because the same inputs give the same answer. Clinical agents from the Agent Catalog can work over that graph on-premises, so patient details stay inside the hospital network.
How we’d model it
Presenting finding—indicates→Urgency levelUrgency level—definedBy→Society guidelineTriage recommendation—derivedFrom→Presenting findingTriage recommendation—citesSection→Guideline section
- Research Identifies Blind Spots in AI Medical Triage — Mount Sinai mountsinai.org ↗
- ChatGPT Health fails critical emergency and suicide safety tests — News-Medical news-medical.net ↗
See how Aktiver turns raw data into knowledge graphs your AI can reason over — deterministic, traceable, behind your firewall.