PDF herunterladen
Appl Clin Inform 2024; 15(02): 327-334
DOI: 10.1055/a-2272-6184
Research Article

Usability Testing of Situation Awareness Clinical Decision Support in the Intensive Care Unit

Matthew J. Molloy
1   Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
2   Division of Hospital Medicine, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
3   Division of Biomedical Informatics, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
,
Matthew Zackoff
1   Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
4   Division of Critical Care, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
,
Annika Gifford
5   Brigham Young University, Provo, Utah, United States
,
Philip Hagedorn
1   Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
2   Division of Hospital Medicine, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
3   Division of Biomedical Informatics, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
,
Ken Tegtmeyer
1   Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
4   Division of Critical Care, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
,
Maria T. Britto
1   Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
6   James M. Anderson Center for Health Systems Excellence, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
,
Maya Dewan
1   Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
3   Division of Biomedical Informatics, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
4   Division of Critical Care, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
6   James M. Anderson Center for Health Systems Excellence, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
› Institutsangaben Funding Dr. Dewan receives career development and research support from the Agency for Healthcare Research and Quality (K08-HS026975), which supported this study.
› Weitere Informationen
    Lizenzen und Reprints

Abstract

Objective Our objective was to evaluate the usability of an automated clinical decision support (CDS) tool previously implemented in the pediatric intensive care unit (PICU) to promote shared situation awareness among the medical team to prevent serious safety events within children's hospitals.

Methods We conducted a mixed-methods usability evaluation of a CDS tool in a PICU at a large, urban, quaternary, free-standing children's hospital in the Midwest. Quantitative assessment was done using the system usability scale (SUS), while qualitative assessment involved think-aloud usability testing. The SUS was scored according to survey guidelines. For think-aloud testing, task times were calculated, and means and standard deviations were determined, stratified by role. Qualitative feedback from participants and moderator observations were summarized.

Results Fifty-one PICU staff members, including physicians, advanced practice providers, nurses, and respiratory therapists, completed the SUS, while ten participants underwent think-aloud usability testing. The overall median usability score was 87.5 (interquartile range: 80–95), with over 96% rating the tool's usability as “good” or “excellent.” Task completion times ranged from 2 to 92 seconds, with the quickest completion for reviewing high-risk criteria and the slowest for adding to high-risk criteria. Observations and participant responses from think-aloud testing highlighted positive aspects of learnability and clear display of complex information that is easily accessed, as well as opportunities for improvement in tool integration into clinical workflows.

Conclusion The PICU Warning Tool demonstrates good usability in the critical care setting. This study demonstrates the value of postimplementation usability testing in identifying opportunities for continued improvement of CDS tools.

Keywords

clinical decision support - usability testing - critical care

Ethical Approval

Our study was approved by the local Institutional Review Board (2020-0202) and was determined to be nonhuman subjects research.


Supplementary Material



Publikationsverlauf

Eingereicht: 14. September 2023

Angenommen: 18. Februar 2024

Accepted Manuscript online:
20. Februar 2024

Artikel online veröffentlicht:
01. Mai 2024

© 2024. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Burrus S, Hall M, Tooley E, Conrad K, Bettenhausen JL, Kemper C. Factors related to serious safety events in a children's hospital patient safety collaborative. Pediatrics 2021; 148 (03) e2020030346
    CrossrefPubMedGoogle Scholar
  • 2 Endsley MR. Measurement of situation awareness in dynamic systems. Hum Factors 1995; 37: 65-84
    CrossrefPubMedGoogle Scholar
  • 3 Endsley MR. Designing for situation awareness in complex systems. Proceedings of the 2nd International Workshop on Symbiosis of Humans, Artifacts and Environment: 2001
    PubMedGoogle Scholar
  • 4 Bates DW, Kuperman GJ, Wang S. et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530
    CrossrefPubMedGoogle Scholar
  • 5 Kushniruk A, Triola M, Stein B, Borycki E, Kannry J. The relationship of usability to medical error: an evaluation of errors associated with usability problems in the use of a handheld application for prescribing medications. Stud Health Technol Inform 2004; 107 (Pt 2): 1073-1076
    PubMedGoogle Scholar
  • 6 Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform 2005; 74 (7-8): 519-526
    CrossrefPubMedGoogle Scholar
  • 7 Norman DA. The Design of Everyday Things. New York: Basic Books; 2013
    Google Scholar
  • 8 Preece J, Rogers Y, Sharp H. et al. Human–computer interaction. New York: Addison-Wesley Publishing Company; 1994
    Google Scholar
  • 9 Orenstein EW, Boudreaux J, Rollins M. et al. Formative usability testing reduces severe blood product ordering errors. Appl Clin Inform 2019; 10 (05) 981-990
    Artikel in Thieme ConnectPubMedGoogle Scholar
  • 10 Akhloufi H, Verhaegh SJC, Jaspers MWM, Melles DC, van der Sijs H, Verbon A. A usability study to improve a clinical decision support system for the prescription of antibiotic drugs. PLoS One 2019; 14 (09) e0223073
    CrossrefPubMedGoogle Scholar
  • 11 Wright MC, Dunbar S, Macpherson BC. et al. Toward designing information display to support critical care. A qualitative contextual evaluation and visioning effort. Appl Clin Inform 2016; 7 (04) 912-929
    Artikel in Thieme ConnectPubMedGoogle Scholar
  • 12 Bode L, Schamer S, Böhnke J. et al; ELISE Study Group. Tracing the progression of sepsis in critically ill children: clinical decision support for detection of hematologic dysfunction. Appl Clin Inform 2022; 13 (05) 1002-1014
    Artikel in Thieme ConnectPubMedGoogle Scholar
  • 13 Ronan CE, Crable EL, Drainoni M-L, Walkey AJ. The impact of clinical decision support systems on provider behavior in the inpatient setting: a systematic review and meta-analysis. J Hosp Med 2022; 17 (05) 368-383
    CrossrefPubMedGoogle Scholar
  • 14 Dziorny AC, Heneghan JA, Bhat MA. et al; Pediatric Data Science and Analytics (PEDAL) Subgroup of the Pediatric Acute Lung Injury and Sepsis Investigators (PALISI) Network. Clinical decision support in the PICU: implications for design and evaluation. Pediatr Crit Care Med 2022; 23 (08) e392-e396
    CrossrefPubMedGoogle Scholar
  • 15 Wright A, Phansalkar S, Bloomrosen M. et al. Best practices in clinical decision support: the case of preventive care reminders. Appl Clin Inform 2010; 1 (03) 331-345
    Artikel in Thieme ConnectPubMedGoogle Scholar
  • 16 Shelov E, Muthu N, Wolfe H. et al. Design and implementation of a pediatric ICU acuity scoring tool as clinical decision support. Appl Clin Inform 2018; 9 (03) 576-587
    Artikel in Thieme ConnectPubMedGoogle Scholar
  • 17 Dewan M, Muthu N, Shelov E. et al. Performance of a clinical decision support tool to identify PICU patients at high risk for clinical deterioration. Pediatr Crit Care Med 2020; 21 (02) 129-135
    CrossrefPubMedGoogle Scholar
  • 18 Dewan M, Soberano B, Sosa T. et al. Assessment of a situation awareness quality improvement intervention to reduce cardiac arrests in the PICU. Pediatr Crit Care Med 2022; 23 (01) 4-12
    CrossrefPubMedGoogle Scholar
  • 19 Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform 2012; 45 (06) 1202-1216
    CrossrefPubMedGoogle Scholar
  • 20 Brooke J. SUS: a retrospective. J Usability Stud 2013; 8 (02) 29-40
    PubMedGoogle Scholar
  • 21 Press A, McCullagh L, Khan S, Schachter A, Pardo S, McGinn T. Usability testing of a complex clinical decision support tool in the emergency department: lessons learned. JMIR Human Factors 2015; 2 (02) e14
    CrossrefPubMedGoogle Scholar
  • 22 Nielsen J, Landauer TK. A mathematical model of the finding of usability problems. In: Proceedings of the INTERCHI'93 Conference on Human Factors in Computing Systems. Amsterdam, The Netherlands, The Netherlands: IOS Press 1993: 206-213
    PubMedGoogle Scholar
  • 23 Faulkner L. Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Methods Instrum Comput 2003; 35 (03) 379-383
    CrossrefPubMedGoogle Scholar
  • 24 Brooke J. SUS—a quick and dirty usability scale. In: Jordan PW, Thomas B, McClelland IL, Weerdmeester B. eds. Usability Evaluation in Industry. England: Routeledge; 1996
    Google Scholar
  • 25 Britto MT, Jimison HB, Munafo JK, Wissman J, Rogers ML, Hersh W. Usability testing finds problems for novice users of pediatric portals. J Am Med Inform Assoc 2009; 16 (05) 660-669
    CrossrefPubMedGoogle Scholar
  • 26 Nguyen TH, Cunha PP, Rowland AF, Orenstein E, Lee T, Kandaswamy S. User-centered design and evaluation of clinical decision support to improve early peanut introduction: formative study. JMIR Form Res 2023; 7: e47574
    CrossrefPubMedGoogle Scholar
  • 27 Kushniruk AW, Borycki EM, Kuwata S, Kannry J. Emerging approaches to usability evaluation of health information systems: towards in-situ analysis of complex healthcare systems and environments. Stud Health Technol Inform 2011; 169: 915-919
    PubMedGoogle Scholar
  • 28 Patel VL, Kannampallil TG. Cognitive informatics in biomedicine and healthcare. J Biomed Inform 2015; 53: 3-14
    CrossrefPubMedGoogle Scholar
  • 29 Dewan M, Vidrine R, Zackoff M. et al. Design, implementation, and validation of a pediatric ICU sepsis prediction tool as clinical decision support. Appl Clin Inform 2020; 11 (02) 218-225
    Artikel in Thieme ConnectPubMedGoogle Scholar
  • 30 Martin B, Mulhern B, Majors M. et al. Improving pediatric intensive care unit discharge timeliness of infants with bronchiolitis using clinical decision support. Appl Clin Inform 2023; 14 (02) 392-399
    Artikel in Thieme ConnectPubMedGoogle Scholar
  • 31 Wulff A, Montag S, Marschollek M, Jack T. Clinical decision-support systems for detection of systemic inflammatory response syndrome, sepsis, and septic shock in critically ill patients: a systematic review. Methods Inf Med 2019; 58 (S 02): e43-e57
    Artikel in Thieme ConnectPubMedGoogle Scholar
  • 32 Chen JH, Asch SM. Machine learning and prediction in medicine—beyond the peak of inflated expectations. N Engl J Med 2017; 376 (26) 2507-2509
    CrossrefPubMedGoogle Scholar
  • 33 Park J, Zhong X, Dong Y, Barwise A, Pickering BW. Investigating the cognitive capacity constraints of an ICU care team using a systems engineering approach. BMC Anesthesiol 2022; 22 (01) 10
    CrossrefPubMedGoogle Scholar