This week the Auditor General of Ontario released a special report on the provincial government’s use of artificial intelligence. Its most striking section covers AI scribes: the tools that listen to a clinician’s conversation with a patient and draft the medical note. In tests run during procurement, every one of the 20 vendors that Supply Ontario went on to approve produced notes with at least one type of inaccuracy.
Nine of the 20 systems fabricated information. Twelve recorded a different drug from the one the doctor prescribed. Seventeen missed key details about patients’ mental health in at least one of two tests. And in the scoring used to select vendors, the accuracy of the generated notes was worth 4% of the total.
/ 01What the auditor found
The AI Scribe programme was initiated and partly funded by Ontario’s Ministry of Health for physicians, nurse practitioners and other health professionals. Supply Ontario, the provincial procurement agency, set up a vendor-of-record arrangement, designed with OntarioMD, Ontario Health and the ministry, so providers could buy from pre-qualified vendors.
As part of the evaluation, all bidders received the same two simulated doctor–patient recordings, and medical professionals compared the generated notes with what was actually said. The report groups the problems into three types:
- Hallucinations. Nine systems added treatment steps that were never discussed, such as a therapy referral or blood tests. Evaluators also flagged invented clinical statements, including notes saying there were “no masses found” or that the patient was anxious, when neither came up in the recording.
- Incorrect information. Twelve systems (60%) captured a different drug from the one prescribed.
- Missing information. Seventeen systems (85%) missed key mental health details in at least one test; six missed them fully or partially across both.
The scoring explains part of how these systems were still approved. Domestic presence in Ontario carried 30% of the available points. Accuracy carried 4%, bias controls 2%, and threat-risk and privacy impact assessments 2%. There were no minimum passing scores for those criteria, so a vendor could in principle score zero on accuracy and still qualify. The report also found that 11 of the 20 approved vendors submitted no third-party audit reports or ISO 27001 certification, and five did not submit required threat-risk or privacy impact assessments.
OntarioMD has told doctors to review the generated notes manually, but the auditor noted that clinicians were not required to attest that they had done so through a sign-off feature. It contrasted this with England, where NHS guidance from April 2025 directs organisations to use only AI scribes registered with the MHRA as Class I medical devices.
/ 02The responses
Supply Ontario agreed with the recommendations. It said it will review guidance from Ontario’s Information and Privacy Commissioner and work with vendors to “determine the feasibility” of mandatory confirmation that notes were reviewed. A ministry spokesperson told CBC, as reported by The Register, that more than 5,000 Ontario physicians participate in the programme and that there are “no known reports of patient harms associated with the technology.”
Some in the sector pushed back. Canadian Healthcare Technology reported that consultant Will Falk pointed out the tests were run on earlier-generation systems and do not describe current use, while family physician Dr. Paul Forman said scribes reduce burnout but must still be checked for accuracy. Auditor General Shelley Spence’s advice to patients was simple: “Please look at the transcript when you’re done.”
/ 03Why this is a design problem, not just a procurement one
It is tempting to read the report as a story about scoring weights, and the weights clearly need fixing. But the deeper issue is that a free-text summary from a generative model gives the clinician nothing to check against except their own memory of the visit. A fabricated “no masses found” reads exactly like a real one.
Clinical notes are, underneath, structured facts: a medication with a dose, a finding, a referral, a symptom, each said at a particular point in the conversation. When a system extracts those facts into an explicit structure and links each one to the moment in the transcript that supports it, a reviewer can see at a glance which statements have a source and which do not. An unsupported referral becomes visible as unsupported, rather than blending into fluent prose. A prescribed drug can be checked against a formulary or the order itself rather than accepted as transcribed.
That is the general principle the report points toward: verification has to be part of the output, not an instruction to the user. Asking busy clinicians to re-listen to every encounter does not scale. Showing them the evidence behind each line of the note does.
A note that cannot show where each fact came from is a draft, not a record.
/ 04What health systems should take from it
- Test on your own scenarios. The auditor found vendors generated their test notes offline and recommended mandatory live demonstrations in future AI procurements. Include medication changes and mental health content, where the errors clustered.
- Set floors, not just weights. Accuracy, security and bias controls need minimum scores that a vendor cannot average away.
- Require traceability in the product. Ask vendors to show which part of the conversation supports each element of the note, and to enforce clinician sign-off.
- Treat structured data as the deliverable. Medications, findings and orders should come out as checkable fields, not only as narrative text.
The Ontario report is a reminder that in clinical documentation, the source link is not a nice-to-have. It is what makes a generated note safe to sign.
The auditor’s numbers describe tools that write fluent prose with no receipts. Weighting accuracy at 4% was a mistake, but even a perfect score sheet would not tell a physician which line of today’s note was invented.
A scribe built on a clinical graph would emit statements rather than paragraphs: a medication with dose and route, a finding, a referral, each linked to the transcript segment that supports it and checked against the formulary and the order actually placed. A referral nobody mentioned arrives with no supporting segment and is flagged before sign-off. The reviewer’s job changes from rereading the whole note to accepting or deleting the unsupported statements, and the sign-off becomes an attestation tied to specific facts rather than a click. The same graph can carry the guideline content behind a plan of care, as in our cardiovascular build of 145 concepts and 284 recommended actions, and running on-premises keeps recordings and PHI inside the clinic.
How we’d model it
Note statement—derivedFrom→Transcript segmentMedication statement—checkedAgainst→Medication orderClinical note—attestedBy→Clinician
- Use of Artificial Intelligence in the Ontario Government (Special Report 2026) — Office of the Auditor General of Ontario auditor.on.ca ↗
- Sick and wrong: Ontario auditors find doctors' AI note takers routinely blow basic facts — The Register theregister.com ↗
- Ontario AG finds flaws in AI scribes — Canadian Healthcare Technology canhealth.com ↗
See how Aktiver turns raw data into knowledge graphs your AI can reason over — deterministic, traceable, behind your firewall.