PDF herunterladen
Appl Clin Inform 2022; 13(05): 1024-1032
DOI: 10.1055/s-0042-1757923
Research Article

Real-Time User Feedback to Support Clinical Decision Support System Improvement

David Rubins
1   Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, United States
2   Digital, Mass General Brigham, Boston, Massachusetts, United States
,
Allison B. McCoy
3   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Sayon Dutta
1   Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, United States
2   Digital, Mass General Brigham, Boston, Massachusetts, United States
4   Department of Emergency Medicine, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts, United States
,
Dustin S. McEvoy
2   Digital, Mass General Brigham, Boston, Massachusetts, United States
,
Lorraine Patterson
5   HealthIT, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Amy Miller
1   Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, United States
2   Digital, Mass General Brigham, Boston, Massachusetts, United States
,
John G. Jackson
5   HealthIT, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Gianna Zuccotti
1   Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, United States
2   Digital, Mass General Brigham, Boston, Massachusetts, United States
,
Adam Wright
3   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
› Institutsangaben
› Weitere Informationen
Auch verfügbar auf
    Lizenzen und Reprints

Abstract

Objectives To improve clinical decision support (CDS) by allowing users to provide real-time feedback when they interact with CDS tools and by creating processes for responding to and acting on this feedback.

Methods Two organizations implemented similar real-time feedback tools and processes in their electronic health record and gathered data over a 30-month period. At both sites, users could provide feedback by using Likert feedback links embedded in all end-user facing alerts, with results stored outside the electronic health record, and provide feedback as a comment when they overrode an alert. Both systems are monitored daily by clinical informatics teams.

Results The two sites received 2,639 Likert feedback comments and 623,270 override comments over a 30-month period. Through four case studies, we describe our use of end-user feedback to rapidly respond to build errors, as well as identifying inaccurate knowledge management, user-interface issues, and unique workflows.

Conclusion Feedback on CDS tools can be solicited in multiple ways, and it contains valuable and actionable suggestions to improve CDS alerts. Additionally, end users appreciate knowing their feedback is being received and may also make other suggestions to improve the electronic health record. Incorporation of end-user feedback into CDS monitoring, evaluation, and remediation is a way to improve CDS.

Keywords

clinical decision support - feedback - CDS monitoring

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects. This project was undertaken as a quality improvement initiative at Mass General Brigham, and as such was not formally supervised by the Institutional Review Board per their policies.




Publikationsverlauf

Eingereicht: 28. März 2022

Angenommen: 13. September 2022

Artikel online veröffentlicht:
26. Oktober 2022

© 2022. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Smith DH, Perrin N, Feldstein A. et al. The impact of prescribing safety alerts for elderly persons in an electronic medical record: an interrupted time series evaluation. Arch Intern Med 2006; 166 (10) 1098-1104
    CrossrefPubMedSuche in Google Scholar
  • 2 Bright TJ, Wong A, Dhurjati R. et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (01) 29-43
    Suche in Google Scholar
  • 3 Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003; 163 (12) 1409-1416
    CrossrefPubMedSuche in Google Scholar
  • 4 McGinn TG, McCullagh L, Kannry J. et al. Efficacy of an evidence-based clinical decision support in primary care practices: a randomized clinical trial. JAMA Intern Med 2013; 173 (17) 1584-1591
    CrossrefPubMedSuche in Google Scholar
  • 5 Chaudhry B, Wang J, Wu S. et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144 (10) 742-752
    Suche in Google Scholar
  • 6 Wright A, Sittig DF, Ash JS. et al. Lessons learned from implementing service-oriented clinical decision support at four sites: a qualitative study. Int J Med Inform 2015; 84 (11) 901-911
    CrossrefPubMedSuche in Google Scholar
  • 7 Bates DW, Cohen M, Leape LL, Overhage JM, Shabot MM, Sheridan T. Reducing the frequency of errors in medicine using information technology. J Am Med Inform Assoc 2001; 8 (04) 299-308
    CrossrefPubMedSuche in Google Scholar
  • 8 Bates DW, Gawande AA. Improving safety with information technology. N Engl J Med 2003; 348 (25) 2526-2534
    CrossrefPubMedSuche in Google Scholar
  • 9 Kwan JL, Lo L, Ferguson J. et al. Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials. BMJ 2020; 370: m3216
    CrossrefPubMedSuche in Google Scholar
  • 10 Stephens AB, Wynn CS, Hofstetter AM. et al. Effect of electronic health record reminders for routine immunizations and immunizations needed for chronic medical conditions. Appl Clin Inform 2021; 12 (05) 1101-1109
    Thieme ConnectPubMedSuche in Google Scholar
  • 11 Stettner S, Adie S, Hanigan S, Thomas M, Pogue K, Zimmerman C. Effect of replacing vendor QTc alerts with a custom QTc risk alert in inpatients. Appl Clin Inform 2022; 13 (01) 19-29
    Thieme ConnectPubMedSuche in Google Scholar
  • 12 Nanji KC, Seger DL, Slight SP. et al. Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc 2018; 25 (05) 476-481
    CrossrefPubMedSuche in Google Scholar
  • 13 Stone EG. Unintended adverse consequences of a clinical decision support system: two cases. J Am Med Inform Assoc 2018; 25 (05) 564-567
    CrossrefPubMedSuche in Google Scholar
  • 14 Ash JS, Sittig DF, Campbell EM, Guappone KP, Dykstra RH. Some unintended consequences of clinical decision support systems. AMIA Annu Symp Proc 2007; 26-30
    PubMedSuche in Google Scholar
  • 15 Wright A, Ai A, Ash J. et al. Clinical decision support alert malfunctions: analysis and empirically derived taxonomy. J Am Med Inform Assoc 2018; 25 (05) 496-506
    CrossrefPubMedSuche in Google Scholar
  • 16 Kassakian SZ, Yackel TR, Gorman PN, Dorr DA. Clinical decisions support malfunctions in a commercial electronic health record. Appl Clin Inform 2017; 8 (03) 910-923
    Thieme ConnectPubMedSuche in Google Scholar
  • 17 Olakotan OO, Yusof MM. Evaluating the alert appropriateness of clinical decision support systems in supporting clinical workflow. J Biomed Inform 2020; 106 (103453): 103453
    CrossrefPubMedSuche in Google Scholar
  • 18 Bates DW, Kuperman GJ, Wang S. et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530
    CrossrefPubMedSuche in Google Scholar
  • 19 Payne TH, Hines LE, Chan RC. et al. Recommendations to improve the usability of drug-drug interaction clinical decision support alerts. J Am Med Inform Assoc 2015; 22 (06) 1243-1250
    CrossrefPubMedSuche in Google Scholar
  • 20 Kastner M, Lottridge D, Marquez C, Newton D, Straus SE. Usability evaluation of a clinical decision support tool for osteoporosis disease management. Implement Sci 2010; 5: 96
    CrossrefPubMedSuche in Google Scholar
  • 21 Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform 2012; 45 (06) 1202-1216
    CrossrefPubMedSuche in Google Scholar
  • 22 Wright A, Ash JS, Aaron S. et al. Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: results of a Delphi study. Int J Med Inform 2018; 118: 78-85
    CrossrefPubMedSuche in Google Scholar
  • 23 Yoshida E, Fei S, Bavuso K, Lagor C, Maviglia S. The value of monitoring clinical decision support interventions. Appl Clin Inform 2018; 9 (01) 163-173
    Thieme ConnectPubMedSuche in Google Scholar
  • 24 Richardson S, Feldstein D, McGinn T. et al. Live usability testing of two complex clinical decision support tools: observational study. JMIR Human Factors 2019; 6 (02) e12471
    CrossrefPubMedSuche in Google Scholar
  • 25 Wright A, Hickman TTT, McEvoy D. et al. Analysis of clinical decision support system malfunctions: a case series and survey. J Am Med Inform Assoc 2016; 23 (06) 1068-1076
    CrossrefPubMedSuche in Google Scholar
  • 26 Lam JH, Ng O. Monitoring clinical decision support in the electronic health record. Am J Health Syst Pharm 2017; 74 (15) 1130-1133
    CrossrefPubMedSuche in Google Scholar
  • 27 Liu S, Wright A, Hauskrecht M. Change-point detection method for clinical decision support system rule monitoring. Artif Intell Med 2018; 91: 49-56
    CrossrefPubMedSuche in Google Scholar
  • 28 Aaron S, McEvoy DS, Ray S, Hickman TTT, Wright A. Cranky comments: detecting clinical decision support malfunctions through free-text override reasons. J Am Med Inform Assoc 2019; 26 (01) 37-43
    CrossrefPubMedSuche in Google Scholar
  • 29 Chaparro JD, Hussain C, Lee JA, Hehmeyer J, Nguyen M, Hoffman J. Reducing interruptive alert burden using quality improvement methodology. Appl Clin Inform 2020; 11 (01) 46-58
    Suche in Google Scholar
  • 30 Orenstein EW, Muthu N, Weitkamp AO. et al. Towards a maturity model for clinical decision support operations. Appl Clin Inform 2019; 10 (05) 810-819
    Thieme ConnectPubMedSuche in Google Scholar
  • 31 Jones BE, Collingridge DS, Vines CG. et al. CDS in a learning health care system: Identifying physicians' reasons for rejection of best-practice recommendations in pneumonia through computerized clinical decision support. Appl Clin Inform 2019; 10 (01) 1-9
    Thieme ConnectPubMedSuche in Google Scholar
  • 32 Pfistermeister B, Sedlmayr B, Patapovas A. et al. Development of a standardized rating tool for drug alerts to reduce information overload. Methods Inf Med 2016; 55 (06) 507-515
    Thieme ConnectPubMedSuche in Google Scholar
  • 33 Yoo J, Lee J, Rhee PL. et al. Alert override patterns with a medication clinical decision support system in an academic emergency department: retrospective descriptive study. JMIR Med Inform 2020; 8 (11) e23351
    CrossrefPubMedSuche in Google Scholar
  • 34 Osheroff J, Teich J, Levick D. et al. Improving Outcomes with Clinical Decision Support. 2nd ed. Chicago, IL: HIMSS Publishing; 2012
    CrossrefSuche in Google Scholar
  • 35 Wright A, Wright AP, Aaron S, Sittig DF. Smashing the strict hierarchy: three cases of clinical decision support malfunctions involving carvedilol. J Am Med Inform Assoc 2018; 25 (11) 1552-1555
    CrossrefPubMedSuche in Google Scholar
  • 36 Ai A, Wong A, Amato M, Wright A. Communication failure: analysis of prescribers' use of an internal free-text field on electronic prescriptions. J Am Med Inform Assoc 2018; 25 (06) 709-714
    CrossrefPubMedSuche in Google Scholar