TL;DR
An Ontario audit reveals that 9 out of 20 AI note-taking systems for healthcare providers frequently produce inaccurate or fabricated information. The findings highlight concerns about AI reliability in critical medical documentation. The evaluation process and scoring criteria are under scrutiny.
The Ontario Office of the Auditor General has found that nine out of 20 AI-based medical note systems approved for use in the province routinely produce inaccurate, fabricated, or incomplete information, raising concerns about their reliability in healthcare settings.
The audit evaluated 20 AI Scribe systems used by Ontario healthcare providers, including physicians and nurse practitioners. The evaluation involved simulated doctor-patient recordings, which were reviewed alongside AI-generated notes. The findings reveal that nearly half of these systems fabricated information, such as incorrectly suggesting that no masses were found in a patient or that the patient was anxious, despite these details not being discussed in the recordings.
Additionally, 12 of the 20 systems inserted incorrect medication data into patient records, and 17 missed key mental health details discussed during consultations. Six systems either partially or fully overlooked mental health issues. The report indicates that these inaccuracies could have serious implications for patient safety and treatment decisions.
Why It Matters
This report underscores the potential risks of deploying AI systems in critical healthcare documentation, where inaccuracies could lead to misdiagnoses, inappropriate treatments, or patient harm. The findings also raise questions about the adequacy of current evaluation and oversight processes for AI tools in medical settings, especially given the minimal weight assigned to accuracy and safety in the vendor scoring system.
Plaud Note AI Voice Recorder, Note Taker w/Case, App Control, Transcribe & Summarize with AI, Support 112 Languages, for Meetings, Calls, Lectures, Professionals, Teams, Black, Non-Pro Version
Plaud Intelligence: Capture conversations in 112 languages and generate accurate transcripts with the Plaud App and Web. Plaud…
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Background
The Ontario AI Scribe program was initiated to streamline clinical documentation, with over 5,000 physicians participating. The evaluation process involved a scoring system where only 4 percent of the total score was based on accuracy, and other safety-related criteria like bias controls and security measures accounted for even less. Critics argue that this evaluation framework may favor vendors with a domestic presence over those with more reliable technology, potentially leading to the selection of subpar AI tools.
Previous studies have shown that large language models often fail in medical contexts, with some research indicating an 80 percent rate of inadequate differential diagnoses. The current report suggests that similar issues may be present in AI note-taking systems used in Ontario, emphasizing the need for stricter oversight.
“Nine out of 20 evaluated AI systems fabricated information and made suggestions to treatment plans that were not discussed in the recordings.”
— Ontario Office of the Auditor General
“Doctors are advised to manually review AI-generated notes for accuracy, but currently, no mandatory attestation feature exists in these systems.”
— OntarioMD
AI for Therapists: The Practical Guide to HIPAA-Compliant AI Tools, Prompt Engineering, and Ethical Workflows for Mental Health Professionals (AI for Professionals)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What Remains Unclear
It is still unclear how widespread these inaccuracies are in actual clinical practice, as the report is based on simulated evaluations. The Ontario Ministry of Health has not confirmed whether reforms or stricter oversight will be implemented following the findings, and no reports of patient harm have been publicly linked to the current AI systems.
AI in Clinical Medicine: A Practical Guide for Healthcare Professionals
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What’s Next
The Ontario government and the AI vendors involved are expected to review the audit findings. Future steps may include revising evaluation criteria to prioritize accuracy and safety, implementing mandatory review protocols for AI-generated notes, and conducting further real-world assessments to verify the systems’ performance.
Physical Therapy Brain Sheet Notepad – Inpatient Clinical Documentation Template for PT Students & Clinicians | Never Miss Important Details, Stay Organized & Improve Charting | 8 Patients Per Sheet (50 Sheets)
🧠 NEVER MISS IMPORTANT PATIENT DETAILS Track medical history, precautions, surgeries, and assistive devices in one place so…
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What specific inaccuracies were found in the AI systems?
The audit found that 9 of 20 systems fabricated information, such as false treatment suggestions, and 12 inserted incorrect medication data. Many also missed key mental health details discussed during consultations.
Are these AI note-takers currently used in real clinical settings?
Yes, more than 5,000 physicians in Ontario are participating in the AI Scribe program, though the findings are based on simulated evaluations rather than actual patient records.
What are the risks of using inaccurate AI in healthcare documentation?
Inaccurate notes can lead to misdiagnoses, inappropriate treatments, and potential patient harm, raising concerns about the safety and reliability of AI tools in critical medical decisions.
Will the Ontario government change its evaluation process for AI vendors?
The report criticizes the current scoring criteria, which give minimal weight to accuracy and safety. Future reforms are likely, but specific plans have not yet been announced.
