Dashboarding to Monitor Machine-Learning-Based Clinical Decision Support Interventions

Abstract

Background Existing monitoring of machine-learning-based clinical decision support (ML-CDS) is focused predominantly on the ML outputs and accuracy thereof. Improving patient care requires not only accurate algorithms but also systems of care that enable the output of these algorithms to drive specific actions by care teams, necessitating expanding their monitoring.

Objectives In this case report, we describe the creation of a dashboard that allows the intervention development team and operational stakeholders to govern and identify potential issues that may require corrective action by bridging the monitoring gap between model outputs and patient outcomes.

Methods We used an iterative development process to build a dashboard to monitor the performance of our intervention in the broader context of the care system.

Results Our investigation of best practices elsewhere, iterative design, and expert consultation led us to anchor our dashboard on alluvial charts and control charts. Both the development process and the dashboard itself illuminated areas to improve the broader intervention.

Conclusion We propose that monitoring ML-CDS algorithms with regular dashboards that allow both a context-level view of the system and a drilled down view of specific components is a critical part of implementing these algorithms to ensure that these tools function appropriately within the broader care system.

Keywords

dashboards - machine learning - clinical decision support - clinical informatics - statistical process control

Protection of Human and Animal Subjects

This larger intervention has been deemed minimal risk by our institutional review board and is registered at clinicaltrials.gov as NCT05810064; https://www.clinicaltrials.gov/study/NCT05810064.

Publication History

Received: 22 August 2023

Accepted: 28 November 2023

Accepted Manuscript online:
29 November 2023

Article published online:
28 February 2024

• References

• 1 Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff (Millwood) 2014; 33 (07) 1123-1131
• 2 Chen JH, Asch SM. Machine learning and prediction in medicine: beyond the peak of inflated expectations. N Engl J Med 2017; 376 (26) 2507-2509
• 3 Obermeyer Z, Emanuel EJ. Predicting the future: big data, machine learning, and clinical medicine. N Engl J Med 2016; 375 (13) 1216-1219
• 4 Brewer LC, Fortuna KL, Jones C. et al. Back to the future: achieving health equity through health informatics and digital health. JMIR Mhealth Uhealth 2020; 8 (01) e14512
• 5 Berlyand Y, Raja AS, Dorner SC. et al. How artificial intelligence could transform emergency department operations. Am J Emerg Med 2018; 36 (08) 1515-1517
• 6 Salwei ME, Carayon P, Hoonakker PLT. et al. Workflow integration analysis of a human factors-based clinical decision support in the emergency department. Appl Ergon 2021; 97: 103498
• 7 Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff (Millwood) 2008; 27 (03) 759-769
• 8 Sikka R, Morath JM, Leape L. The quadruple aim: care, health, cost and meaning in work. BMJ Qual Saf 2015; 24 (10) 608-610
• 9 Nundy S, Cooper LA, Mate KS. The quintuple aim for health care improvement: a new imperative to advance health equity. JAMA 2022; 327 (06) 521-522
• 10 Carayon P, Wooldridge A, Hoonakker P, Hundt AS, Kelly MM. SEIPS 3.0: Human-centered design of the patient journey for patient safety. Appl Ergon 2020; 84: 103033
• 11 Jacobsohn GC, Leaf M, Liao F. et al. Collaborative design and implementation of a clinical decision support system for automated fall-risk identification and referrals in emergency departments. Healthc (Amst) 2022; 10 (01) 100598
• 12 Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17 (01) 36
• 13 Foster M, Albanese C, Chen Q. et al. Heart failure dashboard design and validation to improve care of veterans. Appl Clin Inform 2020; 11 (01) 153-159
• 14 Safranek CW, Feitzinger L, Joyner AKC. et al. Visualizing opioid-use variation in a pediatric perioperative dashboard. Appl Clin Inform 2022; 13 (02) 370-379
• 15 Nelson O, Sturgis B, Gilbert K. et al. A visual analytics dashboard to summarize serial anesthesia records in pediatric radiation treatment. Appl Clin Inform 2019; 10 (04) 563-569
• 16 Radhakrishnan K, Monsen KA, Bae SH, Zhang W. Clinical Relevance for Quality Improvement. Visual analytics for pattern discovery in home care. Appl Clin Inform 2016; 7 (03) 711-730
• 17 Singer SJ, Kellogg KC, Galper AB, Viola D. Enhancing the value to users of machine learning-based clinical decision support tools: A framework for iterative, collaborative development and implementation. Health Care Manage Rev 2022; 47 (02) E21-E31
• 18 Duckworth C, Chmiel FP, Burns DK. et al. Using explainable machine learning to characterise data drift and detect emergent health risks for emergency department admissions during COVID-19. Sci Rep 2021; 11 (01) 23017
• 19 Mišić VV, Rajaram K, Gabel E. A simulation-based evaluation of machine learning models for clinical decision support: application and analysis using hospital readmission. NPJ Digit Med 2021; 4 (01) 98
• 20 Cieslak D, Chawla N. A framework for monitoring classifiers' performance: When and why failure occurs?. Knowl Inf Syst 2009; 18: 83-108
• 21 Bedoya AD, Economou-Zavlanos NJ, Goldstein BA. et al. A framework for the oversight and local deployment of safe and high-quality prediction models. J Am Med Inform Assoc 2022; 29 (09) 1631-1636
• 22 Feng J, Phillips RV, Malenica I. et al. Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare. NPJ Digit Med 2022; 5 (01) 66
• 23 Patterson BW, Engstrom CJ, Sah V. et al. Training and interpreting machine learning algorithms to evaluate fall risk after emergency department visits. Med Care 2019; 57 (07) 560-566
• 24 Hekman D, Maru AP, Patterson BW. Dashboarding for Machine Learning-based Clinical Decision Support Implementation and Monitoring Toolkit. HIPxChange.org. 2023 . Accessed March 16, 2023 at: https://www.hipxchange.org/CDSDashboard
• 25 Dowding D, Merrill JA. The development of heuristics for evaluation of dashboard visualizations. Appl Clin Inform 2018; 9 (03) 511-518
• 26 Jagannath S, Rifkin RM, Gasparetto CJ. et al; CONNECT MM Registry Investigators. Treatment journeys of patients with newly diagnosed multiple myeloma (NDMM): results from the Connect MM Registry. Clin Lymphoma Myeloma Leuk 2020; 20 (05) 272-276
• 27 Friendly M. Visions and re-visions of Charles Joseph Minard. J Educ Behav Stat 2002; 27 (01) 31-51
• 28 Slyngstad L. The contribution of variable control charts to quality improvement in healthcare: a literature review. J Healthc Leadersh 2021; 13: 221-230
• 29 Pimentel L, Barrueto Jr F. Statistical process control: separating signal from noise in emergency department operations. J Emerg Med 2015; 48 (05) 628-638
• 30 Berwick DM. Controlling variation in health care: a consultation from Walter Shewhart. Med Care 1991; 29 (12) 1212-1225
• 31 Thor J, Lundberg J, Ask J. et al. Application of statistical process control in healthcare improvement: systematic review. Qual Saf Health Care 2007; 16 (05) 387-399
• 32 Sotirovski D. Heuristics for iterative software development. IEEE Softw 2001; 18 (03) 66-73
• 33 Caldiera VRBG, Rombach HD. The goal question metric approach. Encycl Softw Eng 1994; 528-532
• 34 R Core Team. R: A Language and Environment for Statistical Computing. 2020. Accessed December 13, 2023 at: https://www.R-project.org/
• 35 Wickham H. ggplot2: Elegant Graphics for Data Analysis. 2016. Accessed December 13, 2023 at: https://ggplot2.tidyverse.org
• 36 Allaire JJ, Xie Y, McPherson J. et al. rmarkdown: Dynamic Documents for R. 2021. Accessed December 13, 2023 at: https://github.com/rstudio/rmarkdown
• 37 Xie Y, Dervieux C, Riederer E. R Markdown Cookbook. 2020. Accessed December 13, 2023 at: https://bookdown.org/yihui/rmarkdown-cookbook
• 38 Wickham H, Averick M, Bryan J. et al. Welcome to the tidyverse. J Open Source Softw 2019; 4 (43) 1686
• 39 Wilkinson L. The Grammar of Graphics. 2nd ed. New York, NY:: Springer;; 2005
• 40 Fairbanks RJ, Bisantz AM. Understanding better how clinicians work. Ann Emerg Med 2011; 58 (02) 123-125
• 41 Donabedian A. The quality of care. How can it be assessed?. JAMA 1988; 260 (12) 1743-1748
• 42 Carayon P, Schoofs Hundt A, Karsh BT. et al. Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006; 15 (Suppl. 01) i50-i58
• 43 Noyez L. Control charts, Cusum techniques and funnel plots. A review of methods for monitoring performance in healthcare. Interact Cardiovasc Thorac Surg 2009; 9 (03) 494-499
• 44 Hekman DJ, Cochran AL, Maru AP. et al. Effectiveness of an emergency department-based machine learning clinical decision support tool to prevent outpatient falls among older adults: protocol for a quasi-experimental study. JMIR Res Protoc 2023; 12: e48128
• 45 Juran JM. Quality-control handbook. New York, NY: McGraw-Hill; 1951. . Available at: https://cir.nii.ac.jp/crid/1130000795132214656