Safeuristics! Do Heuristic Evaluation Violation Severity Ratings Correlate with Patient Safety Severity Ratings for a Native Electronic Health Record Mobile Application?

Abstract

Objective Usability of electronic health records (EHRs) remains challenging, and poor EHR design has patient safety implications. Heuristic evaluation detects usability issues that can be classified by severity. The National Institute of Standards and Technology provides a safety scale for EHR usability. Our objectives were to investigate the relationship between heuristic severity ratings and safety scale ratings in an effort to analyze EHR safety.

Materials and Methods Heuristic evaluation was conducted on seven common mobile EHR tasks, revealing 58 heuristic violations and 28 unique usability issues. Each usability issue was independently scored for severity by trained hospitalists and a Human Factors researcher and for safety severity by two physician informaticists and two clinical safety professionals.

Results Results demonstrated a positive correlation between heuristic severity and safety severity ratings. Regression analysis demonstrated that 49% of safety risk variability by clinical safety professionals (r = 0.70; n = 28) and 42% of safety risk variability by clinical informatics specialists (r = 0.65; n = 28) was explained by usability severity scoring of problems outlined by heuristic evaluation. Higher severity ratings of the usability issues were associated with increased perceptions of patient safety risk.

Discussion This study demonstrated the use of heuristic evaluation as a technique to quickly identify usability problems in an EHR that could lead to safety issues. Detection of higher severity ratings could help prioritize failures in EHR design that more urgently require design changes. This approach is a cost-effective technique for improving usability while impacting patient safety.

Conclusion Results from this study demonstrate the efficacy of the heuristic evaluation technique to identify usability problems that impact safety of the EHR. Also, the use of interdisciplinary teams for evaluation should be considered for severity assessment.

Keywords

user–computer interface - electronic health records - mobile application - heuristics - safety

• References

• 1 Adler-Milstein J, DesRoches CM, Kralovec P. , et al. Electronic health record adoption in US hospitals: progress continues, but challenges persist. Health Aff (Millwood) 2015; 34 (12) 2174-2180
• 2 Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med 2010; 363 (06) 501-504
• 3 Chaudhry B, Wang J, Wu S. , et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144 (10) 742-752
• 4 Kruse CS, Kristof C, Jones B, Mitchell E, Martinez A. Barriers to electronic health record adoption: a systematic literature review. J Med Syst 2016; 40 (12) 252
• 5 The Office of the National Coordinator for Health Information Technology. Test Procedure for §170.314(g)(3) Safety-enhanced design. Available at: https://www.healthit.gov/sites/default/files/standards-certification/2014-edition-draft-test-procedures/170-314-g-3-safety-enhanced-design-2014-test-procedures-draft-v-1.0.pdf
• 6 Couture B, Lilley E, Chang F. , et al. Applying user-centered design methods to the development of an mHealth application for use in the hospital setting by patients and care partners. Appl Clin Inform 2018; 9 (02) 302-312
• 7 Preece J, Rogers Y, Sharp H. Interaction Design: Beyond Human-Computer Interaction. 4th ed. Chichester: Wiley; 2015
• 8 Sorbello A, Ripple A, Tonning J. , et al. Harnessing scientific literature reports for pharmacovigilance. Prototype software analytical tool development and usability testing. Appl Clin Inform 2017; 8 (01) 291-305
• 9 Tarrell A, Grabenbauer L, McClay J. , et al. Toward improved heuristic evaluation of EHRs. Health Syst (Basingstoke) 2015; 4: 138-150
• 10 Nielsen J, Molich R. Heuristic evaluation of user interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Seattle, Washington, United States, April 1–0, 1990, 249–256. Available at: http://dl.acm.org/citation.cfm?id=97281 . Accessed February 10, 2017
• 11 Hertzum M, Jacobsen NE. The evaluator effect: a chilling fact about usability evaluation methods. Int J Hum Comput Interact 2003; 15: 183-204
• 12 Jeffries R, Desurvire H. Usability testing vs. heuristic evaluation. ACM SIGCHI Bull 1992; 24: 39-41
• 13 Molich R, Nielsen J. Improving a human-computer dialogue. Commun ACM 1990; 33: 338-348
• 14 Zhang J, Patel VL, Johnson TR. , et al. Evaluating and predicting patient safety for medical devices with integral information technology. In: Henriksen K, Battles JB, Marks ES. , et al., ed. Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology). Rockville, MD: Agency for Healthcare Research and Quality; 2005
• 15 Zhang J, Johnson TR, Patel VL, Paige DL, Kubose T. Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform 2003; 36 (1-2): 23-30
• 16 Cooper JB, Newbower RS, Long CD, McPeek B. Preventable anesthesia mishaps: a study of human factors. Anesthesiology 1978; 49 (06) 399-406
• 17 Graham MJ, Kubose TK, Jordan D, Zhang J, Johnson TR, Patel VL. Heuristic evaluation of infusion pumps: implications for patient safety in Intensive Care Units. Int J Med Inform 2004; 73 (11-12): 771-779
• 18 Allen M, Currie LM, Bakken S, Patel VL, Cimino JJ. Heuristic evaluation of paper-based Web pages: a simplified inspection usability methodology. J Biomed Inform 2006; 39 (04) 412-423
• 19 Tang Z, Johnson TR, Tindall RD, Zhang J. Applying heuristic evaluation to improve the usability of a telemedicine system. Telemed J E Health 2006; 12 (01) 24-34
• 20 Bates DW, Leape LL, Cullen DJ. , et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998; 280 (15) 1311-1316
• 21 Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003; 163 (12) 1409-1416
• 22 Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA 2018; 319 (12) 1276-1278
• 23 Walker JM, Carayon P, Leveson N. , et al. EHR safety: the way forward to safe and effective systems. J Am Med Inform Assoc 2008; 15 (03) 272-277
• 24 Han YY, Carcillo JA, Venkataraman ST. , et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116 (06) 1506-1512
• 25 Graber ML, Byrne C, Johnston D. The impact of electronic health records on diagnosis. Diagnosis (Berl) 2017; 4 (04) 211-223
• 26 Sittig DF, Wright A, Ash J. , et al. New unintended adverse consequences of electronic health records. IMIA Yearb 2016; (01) 7-12
• 27 Sittig DF, Ash JS, Singh H. Overview of safer guides. SAFER Electron Health Rec Saf Assur Factors EHR Resil 2015; 20: 153-156
• 28 Sittig DF, Singh H. Toward more proactive approaches to safety in the electronic health record era. Jt Comm J Qual Patient Saf 2017; 43 (10) 540-547
• 29 Lowry S, Quinn M, Ramaiah M. , et al. (NISTIR 7804) Technical Evaluation, Testing and Validation of the Usability of Electronic Health Records. NIST Interagency/Internal Report (NISTIR) - 7804. 2012. Available at: https://www.nist.gov/node/592206 . Accessed February 10, 2017
• 30 Borycki E, Kushniruk A, Carvalho C. A methodology for validating safety heuristics using clinical simulations: identifying and preventing possible technology-induced errors related to using health information systems. Comput Math Methods Med 2013. Doi: 10.1155/2013/526419
• 31 Nielsen J. Severity Ratings for Usability Problems. Nielsen Norman Group. 1995. Available at: https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/ . Accessed April 2, 2018
• 32 Ellsworth MA, Dziadzko M, O'Horo JC, Farrell AM, Zhang J, Herasevich V. An appraisal of published usability evaluations of electronic health records via systematic review. J Am Med Inform Assoc 2017; 24 (01) 218-226