Aktiver — semantic AI platform Deterministic · traceable · behind your firewall
Insights / Healthcare

Study finds specialised clinical AI tools trail general-purpose models

A study published this month in Nature Medicine by researchers at NYU Langone Health and the University of Texas at Austin compared two specialised clinical AI tools, OpenEvidence and UpToDate Expert AI, with three general-purpose models: OpenAI’s GPT-5.2, Google’s Gemini 3.1 Pro and Anthropic’s Claude Opus 4.6. On all three benchmarks the authors used, the general-purpose models came out ahead.

The researchers concluded that the clinical tools were “not superior to frontier models on knowledge, communication or clinical alignment,” while noting that they may carry institutional legitimacy and are likely safe for routine use.

/ 01What the study measured

According to TechTarget and heise, the evaluation had three parts:

  • MedQA: 500 USMLE-style exam questions. Gemini scored 97.4%, GPT-5.2 94.2%, Claude 90.2%, OpenEvidence 89.6% and UpToDate Expert AI 88.4%.
  • HealthBench: 500 items scored on a 100-point scale. GPT-5.2 scored 88.0, Gemini 79.3, Claude 77.0, OpenEvidence 62.6 and UpToDate Expert AI 61.3.
  • Real clinical queries: 100 anonymised questions drawn from physicians’ own LLM use, rated in a randomised, blinded review by 12 clinicians on a 1–4 scale. Google Search’s AI Overview was included here. Heise reports average ratings of 3.62 for Gemini, 3.54 for GPT-5.2, 3.52 for Claude, 3.27 for AI Overview, 3.24 for OpenEvidence and 3.17 for UpToDate Expert AI.

Heise reports that the authors found no statistically significant differences in safety, though reviewers often noted incomplete content and safety-relevant omissions for OpenEvidence and AI Overview. UpToDate Expert AI declined to answer 19% of the real-world queries, against 1–3% for the general models. The authors suggest that larger training data and faster development cycles may matter more than medical specialisation, and caution that the work is not a definitive ranking.

Two further points from the coverage are worth noting. The authors observed that the architectures, base models and training pipelines of proprietary clinical tools are not public, which makes independent evaluation of their value and safety harder. And they suggested that providers consider hospital-specific models built on institutional data, used alongside general models for less sensitive tasks.

/ 02Why it matters

Clinical AI products have been sold partly on the promise that a tool trained or tuned on medical literature is safer than a general chatbot. This study does not show that the clinical tools are unsafe. It does show that specialisation, by itself, did not deliver better answers on these tests, and that benchmark accuracy is moving fast enough that last year’s positioning can be overtaken.

For hospitals, that is uncomfortable. Procurement decisions are often made on the strength of a product’s clinical branding and content licensing. If the underlying model can be outperformed by a general system months later, and if the vendor’s architecture cannot be inspected, the institution has little basis for judging what it is relying on.

/ 03Accuracy is not the same as verifiability

There is a further gap the benchmarks cannot close. A 97% score on exam questions says how often a model is right on average. It does not tell the clinician in front of a patient whether this particular answer is one of the 3%. What makes an answer usable in practice is whether it can be checked: which guideline it relies on, which version, which recommendation, and whether that recommendation applies to this patient.

Clinical knowledge is unusually well suited to explicit structure. Guidelines are made of conditions, populations, interventions, contraindications and evidence grades, linked by rules. When that knowledge is represented as a graph, with each recommendation tied to the page it came from, an AI system can be required to ground its answer in specific, citable nodes. The clinician then sees not only an answer but the evidence chain behind it, and a missing step or an off-label leap becomes visible.

That approach also speaks to the study’s point about opacity. If the knowledge a system reasons over is explicit and owned by the institution, the choice of language model matters less, and switching to a better one does not mean rebuilding trust from scratch.

A clinical answer is only as useful as the evidence a clinician can check behind it.

/ 04What to watch

  • Vendor responses. Expect clinical AI companies to contest the methodology and publish their own comparisons. Independent, repeatable evaluations matter more than either side’s numbers.
  • Evaluation on local practice. Benchmarks built from real physician queries, like this study’s third test, are more informative than exam questions. Health systems can build their own.
  • Citation quality. Ask any tool, general or specialised, to show which source supports each statement, and test whether the citations actually say what the answer claims.

The study is a useful reminder that the label on a clinical AI tool says little. What it can show you about its reasoning says a lot more.

Aktiver’s take — Healthcare & pharma

The study ranks answers on exams and curated queries. It does not measure what a hospital needs at the bedside: whether a given answer can be checked against the guideline edition the institution has actually adopted.

Represent the guideline as a graph of populations, interventions, contraindications and evidence grades, each recommendation pinned to its section and edition, and the model’s job narrows to selecting and explaining nodes. A physician asking about anticoagulation for a patient with reduced kidney function sees the recommendation, its grade, the exclusion that applies and the source paragraph. A citation that does not support the claim shows up as a missing link rather than being caught weeks later. The authors’ suggestion of hospital-specific models fits naturally: local formularies and protocols join the same graph, so switching from one frontier model to another changes the wording of an answer, not the evidence under it. Clinical agents from the Agent Catalog run over that graph on-premises.

How we’d model it

RecommendationPatient populationEvidence levelGuideline sectionAnswerappliesTohasEvidenceGradesourcedFromcitesRecommendationPatientpopulationEvidencelevelGuidelinesectionAnswerappliesTohasEvidenceGradesourcedFromcites
  • Recommendation —appliesTo→ Patient population
  • Recommendation —hasEvidenceGrade→ Evidence level
  • Recommendation —sourcedFrom→ Guideline section
  • Answer —cites→ Recommendation

See where this applies →

Sources — read the originals
  1. General-purpose AI beats out specialized clinical AI in some assessments — TechTarget techtarget.com ↗
  2. Medicine: Leading LLMs clearly outperform specialized small language models — heise online heise.de ↗
Go deeper

See how Aktiver turns raw data into knowledge graphs your AI can reason over — deterministic, traceable, behind your firewall.

← All insights Subscribe via RSS → Book a demo →
Keep reading