Dashboarding to Monitor Machine-Learning-Based Clinical Decision Support Interventions

Daniel J. Hekman; Hanna J. Barton; Apoorva P. Maru; Graham Wills; Amy L. Cochran; Corey Fritsch; Douglas A. Wiegmann; Frank Liao; Brian W. Patterson

doi:10.1055/a-2219-5175

Applied Clinical Informatics, Inhaltsverzeichnis

Appl Clin Inform 2024; 15(01): 164-169
DOI: 10.1055/a-2219-5175

Case Report

Dashboarding to Monitor Machine-Learning-Based Clinical Decision Support Interventions

Daniel J. Hekman

¹Berbee-Walsh Department of Emergency Medicine, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States

,

Hanna J. Barton

¹Berbee-Walsh Department of Emergency Medicine, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States

,

Apoorva P. Maru

¹Berbee-Walsh Department of Emergency Medicine, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States

,

Graham Wills

²Department of Applied Data Science, UWHealth Hospitals and Clinics, Madison, Wisconsin, United States

,

Amy L. Cochran

³Department of Population Health, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States

,

Corey Fritsch

²Department of Applied Data Science, UWHealth Hospitals and Clinics, Madison, Wisconsin, United States

,

Douglas A. Wiegmann

⁴Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, Wisconsin, United States

,

Frank Liao

²Department of Applied Data Science, UWHealth Hospitals and Clinics, Madison, Wisconsin, United States

,

Brian W. Patterson

¹Berbee-Walsh Department of Emergency Medicine, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States

³Department of Population Health, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States

⁴Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, Wisconsin, United States

› Institutsangaben

Abstract

Background Existing monitoring of machine-learning-based clinical decision support (ML-CDS) is focused predominantly on the ML outputs and accuracy thereof. Improving patient care requires not only accurate algorithms but also systems of care that enable the output of these algorithms to drive specific actions by care teams, necessitating expanding their monitoring.

Objectives In this case report, we describe the creation of a dashboard that allows the intervention development team and operational stakeholders to govern and identify potential issues that may require corrective action by bridging the monitoring gap between model outputs and patient outcomes.

Methods We used an iterative development process to build a dashboard to monitor the performance of our intervention in the broader context of the care system.

Results Our investigation of best practices elsewhere, iterative design, and expert consultation led us to anchor our dashboard on alluvial charts and control charts. Both the development process and the dashboard itself illuminated areas to improve the broader intervention.

Conclusion We propose that monitoring ML-CDS algorithms with regular dashboards that allow both a context-level view of the system and a drilled down view of specific components is a critical part of implementing these algorithms to ensure that these tools function appropriately within the broader care system.

Keywords

dashboards - machine learning - clinical decision support - clinical informatics - statistical process control

Volltext

Referenzen

References
1 Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff (Millwood) 2014; 33 (07) 1123-1131
2 Chen JH, Asch SM. Machine learning and prediction in medicine: beyond the peak of inflated expectations. N Engl J Med 2017; 376 (26) 2507-2509
3 Obermeyer Z, Emanuel EJ. Predicting the future: big data, machine learning, and clinical medicine. N Engl J Med 2016; 375 (13) 1216-1219
4 Brewer LC, Fortuna KL, Jones C. et al. Back to the future: achieving health equity through health informatics and digital health. JMIR Mhealth Uhealth 2020; 8 (01) e14512
5 Berlyand Y, Raja AS, Dorner SC. et al. How artificial intelligence could transform emergency department operations. Am J Emerg Med 2018; 36 (08) 1515-1517
6 Salwei ME, Carayon P, Hoonakker PLT. et al. Workflow integration analysis of a human factors-based clinical decision support in the emergency department. Appl Ergon 2021; 97: 103498
7 Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff (Millwood) 2008; 27 (03) 759-769
8 Sikka R, Morath JM, Leape L. The quadruple aim: care, health, cost and meaning in work. BMJ Qual Saf 2015; 24 (10) 608-610
9 Nundy S, Cooper LA, Mate KS. The quintuple aim for health care improvement: a new imperative to advance health equity. JAMA 2022; 327 (06) 521-522
10 Carayon P, Wooldridge A, Hoonakker P, Hundt AS, Kelly MM. SEIPS 3.0: Human-centered design of the patient journey for patient safety. Appl Ergon 2020; 84: 103033
11 Jacobsohn GC, Leaf M, Liao F. et al. Collaborative design and implementation of a clinical decision support system for automated fall-risk identification and referrals in emergency departments. Healthc (Amst) 2022; 10 (01) 100598
12 Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17 (01) 36
13 Foster M, Albanese C, Chen Q. et al. Heart failure dashboard design and validation to improve care of veterans. Appl Clin Inform 2020; 11 (01) 153-159
14 Safranek CW, Feitzinger L, Joyner AKC. et al. Visualizing opioid-use variation in a pediatric perioperative dashboard. Appl Clin Inform 2022; 13 (02) 370-379
15 Nelson O, Sturgis B, Gilbert K. et al. A visual analytics dashboard to summarize serial anesthesia records in pediatric radiation treatment. Appl Clin Inform 2019; 10 (04) 563-569
16 Radhakrishnan K, Monsen KA, Bae SH, Zhang W. Clinical Relevance for Quality Improvement. Visual analytics for pattern discovery in home care. Appl Clin Inform 2016; 7 (03) 711-730
17 Singer SJ, Kellogg KC, Galper AB, Viola D. Enhancing the value to users of machine learning-based clinical decision support tools: A framework for iterative, collaborative development and implementation. Health Care Manage Rev 2022; 47 (02) E21-E31
18 Duckworth C, Chmiel FP, Burns DK. et al. Using explainable machine learning to characterise data drift and detect emergent health risks for emergency department admissions during COVID-19. Sci Rep 2021; 11 (01) 23017
19 Mišić VV, Rajaram K, Gabel E. A simulation-based evaluation of machine learning models for clinical decision support: application and analysis using hospital readmission. NPJ Digit Med 2021; 4 (01) 98
20 Cieslak D, Chawla N. A framework for monitoring classifiers' performance: When and why failure occurs?. Knowl Inf Syst 2009; 18: 83-108
21 Bedoya AD, Economou-Zavlanos NJ, Goldstein BA. et al. A framework for the oversight and local deployment of safe and high-quality prediction models. J Am Med Inform Assoc 2022; 29 (09) 1631-1636
22 Feng J, Phillips RV, Malenica I. et al. Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare. NPJ Digit Med 2022; 5 (01) 66
23 Patterson BW, Engstrom CJ, Sah V. et al. Training and interpreting machine learning algorithms to evaluate fall risk after emergency department visits. Med Care 2019; 57 (07) 560-566
24 Hekman D, Maru AP, Patterson BW. Dashboarding for Machine Learning-based Clinical Decision Support Implementation and Monitoring Toolkit. HIPxChange.org. 2023 . Accessed March 16, 2023 at: https://www.hipxchange.org/CDSDashboard
25 Dowding D, Merrill JA. The development of heuristics for evaluation of dashboard visualizations. Appl Clin Inform 2018; 9 (03) 511-518
26 Jagannath S, Rifkin RM, Gasparetto CJ. et al; CONNECT MM Registry Investigators. Treatment journeys of patients with newly diagnosed multiple myeloma (NDMM): results from the Connect MM Registry. Clin Lymphoma Myeloma Leuk 2020; 20 (05) 272-276
27 Friendly M. Visions and re-visions of Charles Joseph Minard. J Educ Behav Stat 2002; 27 (01) 31-51
28 Slyngstad L. The contribution of variable control charts to quality improvement in healthcare: a literature review. J Healthc Leadersh 2021; 13: 221-230
29 Pimentel L, Barrueto Jr F. Statistical process control: separating signal from noise in emergency department operations. J Emerg Med 2015; 48 (05) 628-638
30 Berwick DM. Controlling variation in health care: a consultation from Walter Shewhart. Med Care 1991; 29 (12) 1212-1225
31 Thor J, Lundberg J, Ask J. et al. Application of statistical process control in healthcare improvement: systematic review. Qual Saf Health Care 2007; 16 (05) 387-399
32 Sotirovski D. Heuristics for iterative software development. IEEE Softw 2001; 18 (03) 66-73
33 Caldiera VRBG, Rombach HD. The goal question metric approach. Encycl Softw Eng 1994; 528-532
34 R Core Team. R: A Language and Environment for Statistical Computing. 2020. Accessed December 13, 2023 at: https://www.R-project.org/
35 Wickham H. ggplot2: Elegant Graphics for Data Analysis. 2016. Accessed December 13, 2023 at: https://ggplot2.tidyverse.org
36 Allaire JJ, Xie Y, McPherson J. et al. rmarkdown: Dynamic Documents for R. 2021. Accessed December 13, 2023 at: https://github.com/rstudio/rmarkdown
37 Xie Y, Dervieux C, Riederer E. R Markdown Cookbook. 2020. Accessed December 13, 2023 at: https://bookdown.org/yihui/rmarkdown-cookbook
38 Wickham H, Averick M, Bryan J. et al. Welcome to the tidyverse. J Open Source Softw 2019; 4 (43) 1686
39 Wilkinson L. The Grammar of Graphics. 2nd ed. New York, NY:: Springer;; 2005
40 Fairbanks RJ, Bisantz AM. Understanding better how clinicians work. Ann Emerg Med 2011; 58 (02) 123-125
41 Donabedian A. The quality of care. How can it be assessed?. JAMA 1988; 260 (12) 1743-1748
42 Carayon P, Schoofs Hundt A, Karsh BT. et al. Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006; 15 (Suppl. 01) i50-i58
43 Noyez L. Control charts, Cusum techniques and funnel plots. A review of methods for monitoring performance in healthcare. Interact Cardiovasc Thorac Surg 2009; 9 (03) 494-499
44 Hekman DJ, Cochran AL, Maru AP. et al. Effectiveness of an emergency department-based machine learning clinical decision support tool to prevent outpatient falls among older adults: protocol for a quasi-experimental study. JMIR Res Protoc 2023; 12: e48128
45 Juran JM. Quality-control handbook. New York, NY: McGraw-Hill; 1951. . Available at: https://cir.nii.ac.jp/crid/1130000795132214656

Zusatzmaterial

Supplementary Material