A study published by the Journal of the American Medical Informatics Association makes practical suggestions for creating methods, rules, and guidelines to ensure that AI development, testing, supervision, and use in clinical decision support (CDS) systems are done well and safely for patients.

According to the study, AI-enabled Clinical Decision Support (AI-CDS) systems promise to revolutionize healthcare decision-making, necessitating a comprehensive framework for their development, implementation, and regulation, emphasising trustworthiness, transparency, and safety. This framework encompasses various aspects, including model training, explainability, validation, certification, monitoring, and continuous evaluation, while addressing challenges such as data privacy, fairness, and regulatory oversight to ensure responsible integration of AI into clinical workflow.

Materials and Methods

In May 2023, the Division of Clinical Informatics at Beth Israel Deaconess Medical Center and the American Medical Informatics Association co-sponsored a working group on AI in healthcare. In August 2023, there were 4 webinars on AI topics and a 2-day workshop in September 2023 for consensus-building. The event included over 200 industry stakeholders, including clinicians, software developers, academics, ethicists, attorneys, government policy experts, scientists, and patients. The goal was to identify challenges associated with the trusted use of AI-enabled CDS in medical practice. Key issues were identified, and solutions were proposed through qualitative analysis and a 4-month iterative consensus process.

The researchers focused on three specific domains of how AI is being deployed in the healthcare ecosystem: (1) a patient’s viewpoint on how they are using AI in their healthcare journeys, (2) how AI is impacting CDS, and (3) how AI is being used to create, enrich, and use real-world data/evidence.

During the study, the sub-groups developed rapid brainstorming methods and group discussions to develop the core ideas and challenges. The group used a Delphi approach to iterate on the core challenges and the consensus approaches for how they should be prioritized and addressed. 

Key recommendations

Governance of these interconnected issues requires multi-pronged initiatives around transparency, training, and infrastructure from developers, regulators, and healthcare delivery organizations. The study culminated in several key recommendations. Four of them includes:

• Building safe and trustworthy systems;
• Developing validation, verification, and certification processes for AI-CDS systems; 
• Providing a means of safety monitoring and reporting at the national level; and 
• Ensuring that appropriate documentation and end-user training are provided.

Journal of the American Medical Informatics Association