Usability and Acceptability of Clinical Decision Support Based on the KIIDS-TBI Tool for Children with Mild Traumatic Brain Injuries and Intracranial Injuries

Abstract

Background The Kids Intracranial Injury Decision Support tool for Traumatic Brain Injury (KIIDS-TBI) tool is a validated risk prediction model for managing children with mild traumatic brain injuries (mTBI) and intracranial injuries. Electronic clinical decision support (CDS) may facilitate the clinical implementation of this evidence-based guidance.

Objective Our objective was to evaluate the acceptability and usability of an electronic CDS tool for managing children with mTBI and intracranial injuries.

Methods Emergency medicine and neurosurgery physicians (10 each) from 10 hospitals in the United States were recruited to participate in usability testing of a novel CDS prototype in a simulated electronic health record environment. Testing included a think-aloud protocol, an acceptability and usability survey, and a semi-structured interview. The prototype was updated twice during testing to reflect user feedback. Usability problems recorded in the videos were categorized using content analysis. Interview transcripts were analyzed using thematic analysis.

Results Among the 20 participants, most worked at teaching hospitals (80%), freestanding children's hospitals (95%), and level-1 trauma centers (75%). During the two prototype updates, problems with clarity of terminology and navigating through the CDS interface were identified and corrected. Corresponding to these changes, the number of usability problems decreased from 35 in phase 1 to 8 in phase 3 and the number of mistakes made decreased from 18 (phase 1) to 2 (phase 3). Through the survey, participants found the tool easy to use (90%), useful for determining a patient's level of care (95%), and likely to improve resource use (90%) and patient safety (79%). Interview themes related to the CDS's ability to support evidence-based decision-making and improve clinical workflow proposed implementation strategies and potential pitfalls.

Conclusion After iterative evaluation and refinement, the KIIDS-TBI CDS tool was found to be highly usable and useful for aiding the management of children with mTBI and intracranial injuries.

Keywords

clinical decision support - clinical information - traumatic brain injury - children - mixed methods

• References

• 1 National Center for Injury Prevention and Control (US).. Report to Congress on mild traumatic brain injury in the United States: steps to prevent a serious public health problem. Centers for Disease Control and Prevention;; 2003
• 2 McKinlay A, Grace RC, Horwood LJ, Fergusson DM, Ridder EM, MacFarlane MR. Prevalence of traumatic brain injury among children, adolescents and young adults: prospective evidence from a birth cohort. Brain Inj 2008; 22 (02) 175-181
• 3 Schneier AJ, Shields BJ, Hostetler SG, Xiang H, Smith GA. Incidence of pediatric traumatic brain injury and associated hospital resource utilization in the United States. Pediatrics 2006; 118 (02) 483-492
• 4 Koepsell TD, Rivara FP, Vavilala MS. et al. Incidence and descriptive epidemiologic features of traumatic brain injury in King County, Washington. Pediatrics 2011; 128 (05) 946-954
• 5 Bowman SM, Bird TM, Aitken ME, Tilford JM. Trends in hospitalizations associated with pediatric traumatic brain injuries. Pediatrics 2008; 122 (05) 988-993
• 6 Lumba-Brown A, Yeates KO, Sarmiento K. et al. Centers for Disease Control and Prevention Guideline on the Diagnosis and Management of Mild Traumatic Brain Injury Among Children. JAMA Pediatr 2018; 172 (11) e182853
• 7 Kuppermann N, Holmes JF, Dayan PS. et al; Pediatric Emergency Care Applied Research Network (PECARN). Identification of children at very low risk of clinically-important brain injuries after head trauma: a prospective cohort study. Lancet 2009; 374 (9696): 1160-1170
• 8 Babl FE, Borland ML, Phillips N. et al; Paediatric Research in Emergency Departments International Collaborative (PREDICT). Accuracy of PECARN, CATCH, and CHALICE head injury decision rules in children: a prospective cohort study. Lancet 2017; 389 (10087): 2393-2402
• 9 Greenberg JK, Jeffe DB, Carpenter CR. et al. North American survey on the post-neuroimaging management of children with mild head injuries. J Neurosurg Pediatr 2018; 23 (02) 227-235
• 10 Greenberg JK, Stoev IT, Park TS. et al. Management of children with mild traumatic brain injury and intracranial hemorrhage. J Trauma Acute Care Surg 2014; 76 (04) 1089-1095
• 11 Greenberg JK, Yan Y, Carpenter CR. et al. Development and internal validation of a clinical risk score for treating children with mild head trauma and intracranial injury. JAMA Pediatr 2017; 171 (04) 342-349
• 12 Neumayer KE, Sweney J, Fenton SJ, Keenan HT, Flaherty BF. Validation of the “CHIIDA” and application for PICU triage in children with complicated mild traumatic brain injury. J Pediatr Surg 2020; 55 (07) 1255-1259
• 13 Greenberg JK, Ahluwalia R, Hill M. et al. Development and external validation of the KIIDS-TBI tool for managing children with mild traumatic brain injury and intracranial injuries. Acad Emerg Med 2021; 28 (12) 1409-1420
• 14 Stiell IG, Bennett C. Implementation of clinical decision rules in the emergency department. Acad Emerg Med 2007; 14 (11) 955-959
• 15 Green SM. When do clinical decision rules improve patient care?. Ann Emerg Med 2013; 62 (02) 132-135
• 16 Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 2010; 19 (Suppl. 03) i68-i74
• 17 Kiatchai T, Colletti AA, Lyons VH, Grant RM, Vavilala MS, Nair BG. Development and feasibility of a real-time clinical decision support system for traumatic brain injury anesthesia care. Appl Clin Inform 2017; 8 (01) 80-96
• 18 Greenberg JK, Otun A, Nasraddin A. et al. Electronic clinical decision support for children with minor head trauma and intracranial injuries: a sociotechnical analysis. BMC Med Inform Decis Mak 2021; 21 (01) 161
• 19 Nielsen J, Clemmensen T, Yssing C. Getting access to what goes on in people's heads? reflections on the think-aloud technique. Proceedings of the Second Nordic Conference on Human-Computer Interaction; Aarhus, Denmark; 2002
• 20 Yen PY, Bakken S. A comparison of usability evaluation methods: heuristic evaluation versus end-user think-aloud protocol—an example from a web-based communication tool for nurse scheduling. AMIA Annu Symp Proc 2009; 2009: 714-718
• 21 Hartzler AL, Chaudhuri S, Fey BC, Flum DR, Lavallee D. Integrating patient-reported outcomes into spine surgical care through visual dashboards: lessons learned from human-centered design. EGEMS (Wash DC) 2015; 3 (02) 1133
• 22 Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform 2012; 45 (06) 1202-1216
• 23 Jaspers MW, Steen T, van den Bos C, Geenen M. The think aloud method: a guide to user interface design. Int J Med Inform 2004; 73 (11-12): 781-795
• 24 Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform 2009; 78 (05) 340-353
• 25 van den Haak M, De Jong M, Jan Schellens P. Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue. Behav Inf Technol 2003; 22 (05) 339-351
• 26 Brehaut JC, Graham ID, Wood TJ. et al. Measuring acceptability of clinical decision rules: validation of the Ottawa acceptability of decision rules instrument (OADRI) in four countries. Med Decis Making 2010; 30 (03) 398-408
• 27 Yen P-Y, Sousa KH, Bakken S. Examining construct and predictive validity of the Health-IT Usability Evaluation Scale: confirmatory factor analysis and structural equation modeling results. J Am Med Inform Assoc 2014; 21 (e2): e241-e248
• 28 Peres SC, Pham T, Phillips R. Validation of the System Usability Scale (SUS):SUS in the Wild. Proc Hum Factors Ergon Soc Annu Meet 2013; 57 (01) 192-196
• 29 Yen PY, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012; 19 (03) 413-422
• 30 Yen PY, Walker DM, Smith JMG, Zhou MP, Menser TL, McAlearney AS. Usability evaluation of a commercial inpatient portal. Int J Med Inform 2018; 110: 10-18
• 31 Kushniruk AW, Borycki EM. Development of a video coding scheme for analyzing the usability and usefulness of health information systems. Stud Health Technol Inform 2015; 218: 68-73
• 32 Nowell LS, Norris JM, White DE, Moules NJ. Thematic analysis: striving to meet the trustworthiness criteria. Int J Qual Methods 2017; 16 (01) 1609406917733847
• 33 Curry LA, Nembhard IM, Bradley EH. Qualitative and mixed methods provide unique contributions to outcomes research. Circulation 2009; 119 (10) 1442-1452
• 34 Tham E, Swietlik M, Deakyne S. et al; Pediatric Emergency Care Applied Research Network (PECARN). Clinical decision support for a multicenter trial of pediatric head trauma: development, implementation, and lessons learned. Appl Clin Inform 2016; 7 (02) 534-542
• 35 Grinspan ZM, Eldar YC, Gopher D. et al. Guiding Principles for a Pediatric Neurology ICU (neuroPICU) Bedside Multimodal Monitor: findings from an International Working Group. Appl Clin Inform 2016; 7 (02) 380-398
• 36 Gasson S. Human-centered vs. user-centered approaches to information system design. J Inf Technol Theory Appl 2003; 5 (02) 5 (JITTA)
• 37 Masterson Creber RM, Dayan PS, Kuppermann N. et al; Pediatric Emergency Care Applied Research Network (PECARN) and the Clinical Research on Emergency Services and Treatments (CREST) Network. Applying the RE-AIM framework for the evaluation of a clinical decision support tool for pediatric head trauma: a mixed-methods study. Appl Clin Inform 2018; 9 (03) 693-703
• 38 Cabana MD, Rand CS, Powe NR. et al. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA 1999; 282 (15) 1458-1465
• 39 Croskerry P. Clinical cognition and diagnostic error: applications of a dual process model of reasoning. Adv Health Sci Educ Theory Pract 2009; 14 (1, Suppl 1): 27-35
• 40 Ozkaynak M, Metcalf N, Cohen DM, May LS, Dayan PS, Mistry RD. Considerations for designing EHR-embedded clinical decision support systems for antimicrobial stewardship in pediatric emergency departments. Appl Clin Inform 2020; 11 (04) 589-597
• 41 Kane B, Carpenter C. Cognition and decision making. In: The Washington Manual of Patient Safety and Quality Improvement; Philadelphia, PA: Wolters Kluwer; 2016: 195-209
• 42 Stark DE, Kumar RB, Longhurst CA, Wall DP. The quantified brain: a framework for mobile device-based assessment of behavior and neurological function. Appl Clin Inform 2016; 7 (02) 290-298
• 43 Gimbel RW, Pirrallo RG, Lowe SC. et al. Effect of clinical decision rules, patient cost and malpractice information on clinician brain CT image ordering: a randomized controlled trial. BMC Med Inform Decis Mak 2018; 18 (01) 20