Usability Methods for Ensuring Health Information Technology Safety: Evidence-Based Approaches Contribution of the IMIA Working Group Health Informatics for Patient Safety

Summary

Objectives: Issues related to lack of system usability and potential safety hazards continue to be reported in the health information technology (HIT) literature. Usability engineering methods are increasingly used to ensure improved system usability and they are also beginning to be applied more widely for ensuring the safety of HIT applications. These methods are being used in the design and implementation of many HIT systems. In this paper we describe evidence- based approaches to applying usability engineering methods.

Methods: A multi-phased approach to ensuring system usability and safety in healthcare is described. Usability inspection methods are first described including the development of evidence-based safety heuristics for HIT. Laboratory-based usability testing is then conducted under artificial conditions to test if a system has any base level usability problems that need to be corrected. Usability problems that are detected are corrected and then a new phase is initiated where the system is tested under more realistic conditions using clinical simulations. This phase may involve testing the system with simulated patients. Finally, an additional phase may be conducted, involving a naturalistic study of system use under real-world clinical conditions.

Results: The methods described have been employed in the analysis of the usability and safety of a wide range of HIT applications, including electronic health record systems, decision support systems and consumer health applications. It has been found that at least usability inspection and usability testing should be applied prior to the widespread release of HIT. However, wherever possible, additional layers of testing involving clinical simulations and a naturalistic evaluation will likely detect usability and safety issues that may not otherwise be detected prior to widespread system release.

Conclusion: The framework presented in the paper can be applied in order to develop more usable and safer HIT, based on multiple layers of evidence.

Keywords

Usability testing - clinical simulation - technology-induced error - health information technology - patient safety

 

• References

• 1 Borycki EM, Kushniruk AW, Keay L, Kuo A. A framework for diagnosing and identifying where technology-induced errors come from. Stud Health Technol Inform 2009; 37: 56-76.
• 2 Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004; 37: 56-76.
• 3 Preece J, Rogers Y, Sharp H. interaction Design: Beyond Human-Computer Interaction. New York: John Wiley & Sons; 2002
• 4 Borycki E, Kushniruk A, Kuwata S, Kannry J. Use of simulation in the study of clinician workflow. Proc AMIA Annu Symp 2006; p. 61-5.
• 5 Kushniruk AW, Triola M, Borycki E, Stein B, Kannry J. Technology induced error and usability. Int J Med Inform 2005; 74: 519-26.
• 6 Kushniruk A, Borycki E, Kuwata S, Kannry J. Predicting changes in workflow resulting from healthcare information System: ensuring the safety of healthcare. Healthc Q 2006; 9: 114-8.
• 7 Li AC, Kannry JL, Kushniruk A, Chrimes D, McGinn T. Ebonyabo et al. Integrating usability testing and think-aloud protocol analysis with “near-live” clinical simulations in evaluating clinical decision support. Int J Med Inform. 2012 (in press).
• 8 Baylis TB, Kushniruk A, Borycki EM. Low-cost rapid usability testing for health information systems: Is it worth the effort? Proceedings of MIE. 2012. Pisa, Italy.:
• 9 Rasmussen R, Kushniruk A. The long and twisting path: An efficiency evaluation of an electronic whiteboard system. Proceedings of ITCH. 2013. Victoria, Canada: (in press).
• 10 Institute of Medicine. Health IT and patient safety: building safer systems for better care. Committee on Patient Safety and Health Information Technology; 2011. Available from: http://www.nap.edu/catalog.php?record_id=13269
• 11 Ash J, Kilo C, Sahpiro M, Wasserman J, Mc-Mullen C, Hersh W. Roadmap for provision of safer healthcare information systems: Preventing e-Iatrogenesis 2011, report commissioned for the Institute of Medicine..
• 12 Kushniruk A, Borycki E. Low-cost rapid usability engineering: designing and customizing usable healthcare information systems. Healthc Q 2006; 9 (4) 98-102.
• 13 Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE. et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005; 293 (10) 1197-203.
• 14 Magrabi F, Ong M-S, Runciman W, Coiera E. Using FDA reports to inform a classification for health information technology safety problems. J Am Med Inform Assoc 2012; 19: 45-53.
• 15 Borycki EM, Kushniruk AW. Scenario-based testing of health information systems (HIS) in electronic and hybrid environments. Stud Health Technol Inform 2009; 143: 284-89.
• 16 Yui BH, Jim WT, Chen M, Hsu JM, Liu CY, Lee TT. Evaluation of computerized physician order entry system-a satisfaction survey in Taiwan. J Med Syst 2012; 36 (6) 3817-24.
• 17 Magrabi F, Ong M-S, Runciman W, Coiera E. An analysis of computer-related patient safety incidents to inform the development of a classification. J Am Med Inform Assoc 2010; 17: 663-70.
• 18 Borycki EM, Kushniruk AW. Where do technology-induced errors come from? Towards a model for conceptualizing and diagnosing errors caused by technology. In: Human, Social and Organizational Aspects of Health Information Systems. Hershey NY: Idea Group; 2008
• 19 Borycki EM, Kushniruk AW, Bellwood P, Brender J. Technology-induced errors. The current use of frameworks and models from the biomedical and life science literatures. Methods Inf Med 2012; 51 (2) 95-103.
• 20 Borycki EM, Keay E. Methods to assess the safety of health information systems. Healthc Q 2010; 13: 47-52.
• 21 Sittig S, Singh H. Electronic health records and national patient-safety goals. N Engl J Med 2012; 367 (19) 1854-60.
• 22 Kuwata S, Kushniruk A, Borycki E, Watanabe H. Using simulation methods to analyze and predict changes in workflow and potential problems in the use of a bar-coding medication order entry system. AMIA Annu Symp Proc 2006; 994.
• 23 Beuscart-Zephir MC, Pelayo S, Bernonville S. Example of a human factors engineering approach to a medication administration work system: Potential impact on patient safety. Int J Med Inform 2010; 79 (4) 35-42.
• 24 Eslami S, Abu-Hanna A, de Keizer NF. Evaluation of outpatient computerized physician order entry systems: A systematic review. J Am Med Inform Assoc 2007; 14 (4) 400-6.
• 25 IOM Roadmap. Roadmap for provision of safer healthcare information systems: Preventing e-Iat- rogenesis. 2012
• 26 Carvalho CJ, Borycki EM, Kushniruk AW. Ensuring the safety of health information systems: Using heuristics for patient safety. Healthc Q 2009; 12: 49-54.
• 27 Borycki EM, Kushniruk AW, Kuwata S, Kannry J. Use of simulation approaches to the study of clinician workflow. AMIA Annu Symp Proc 2006; 61-5.
• 28 Borycki EM, Kushniruk AW, Anderson J, Anderson M. Designing and integrating clinical and computer-based simulations in health informatics: From real-world to virtual reality. In: Cakaj S. editor. Modeling Simulation and Optimization-Focus on Applications. 2010. p. 31-52. Vukovar: Croatia: In-Teh; http://www.intechopen.com/books/modeling-simulation-and-optimization-focus-on-applications/designing-and-in-tegrating-clinical-and-computer-based-simulations-in-health-informatics-from-real-wo
• 29 Ammenwerth E, Hackl WO, Brinzer K, Christoffersen THE, Jensen S, Lawton K. et al. Simulation studies for the evaluation of health information technologies: Experiences and results. HIM J 2012; 41 (2) 14-20.
• 30 Nielsen J. Usability engineering. New York: Morgan Kaufman; 1993
• 31 Zhang J, Johnson TR, Patel VL, Paige D, Kubose T. Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform 2003; 36 (1-2) 23-30.
• 32 Lewis C, Polson P, Wharton C, Rieman J. Testing a Walkthrough Methodology for Theory-Based Design of Walk-Up-and-Use. Interfaces Chi '90 Proceedings. 1990 p. 235-42.
• 33 Microsoft. Microsoft Health common user interface. 2010 http://www.mscui.net/
• 34 National Health Service. Design for patient safety: Guidelines for safe on-screen display of medication information. 2010 Available from: http://www.nrls.npsa.nhs.uk/resources/?entryid45=66713
• 35 Carvalho C, Borycki EM, Kushniruk AW, Kuwata S, Watanabe H. Simulations to assess medication administration systems. In: Staudinger B, Hoess V, Ostermann H. editors. Nursing and Clinical Informatics: Socio-Technical Approaches. Hershey Pennsylvania: IGI Global; 2009. p. 144-59.
• 36 Borycki EM, Kushniruk AW, Carvalho C. Extending heuristic evaluations to assess safety and workflow aspects of health information systems. 5th International Symposium on Human Factors Engineering in Health Informatics. 2001. Aug 26-27 Trondheim, Norway: Tapir Academic Press.;
• 37 NHS Direct. 2012 http://www.connectingfor-health.nhs.uk/systemsandservices/data/cui
• 38 Kushniruk AW, Bates DW, Bainbridge M, Househ MS, Borycki EM. National efforts to improve health information system safety in Canada, the United States and England. Int J Med Inform. in press.
• 39 Monkman H, Kushniruk A. Appling usability methods to identify health literacy issues: An example using a personal health record. Stud Health Technol Inform 2013; 183: 179-85.