Safeuristics! Do Heuristic Evaluation Violation Severity Ratings Correlate with Patient Safety Severity Ratings for a Native Electronic Health Record Mobile Application?

Brandan Kennedy; Ellen Kerns; Y. Raymond Chan; Barbara S. Chaparro; Sarah D. Fouquet

doi:10.1055/s-0039-1681073

Applied Clinical Informatics, Table of Contents

Appl Clin Inform 2019; 10(02): 210-218
DOI: 10.1055/s-0039-1681073

Research Article

Georg Thieme Verlag KG Stuttgart · New York

Safeuristics! Do Heuristic Evaluation Violation Severity Ratings Correlate with Patient Safety Severity Ratings for a Native Electronic Health Record Mobile Application?

Brandan Kennedy

¹Medical Informatics and Telemedicine, Children's Mercy Hospital, Kansas City, Missouri, United States

²Pediatric Hospital Medicine, Children's Mercy Hospital, Kansas City, Missouri, United States

³Human Factors Collaborative, Children’s Mercy Hospital, Kansas City, Missouri, United States

,

Ellen Kerns

⁴Department of Pediatrics, University of Nebraska Medical Center, Omaha, Nebraska, United States

,

Y. Raymond Chan

²Pediatric Hospital Medicine, Children's Mercy Hospital, Kansas City, Missouri, United States

³Human Factors Collaborative, Children’s Mercy Hospital, Kansas City, Missouri, United States

,

Barbara S. Chaparro

⁵Human Factors and Behavioral Neurobiology, Embry-Riddle Aeronautical University, Daytona Beach, Florida, United States

,

Sarah D. Fouquet

³Human Factors Collaborative, Children’s Mercy Hospital, Kansas City, Missouri, United States

› Author Affiliations

Abstract

Objective Usability of electronic health records (EHRs) remains challenging, and poor EHR design has patient safety implications. Heuristic evaluation detects usability issues that can be classified by severity. The National Institute of Standards and Technology provides a safety scale for EHR usability. Our objectives were to investigate the relationship between heuristic severity ratings and safety scale ratings in an effort to analyze EHR safety.

Materials and Methods Heuristic evaluation was conducted on seven common mobile EHR tasks, revealing 58 heuristic violations and 28 unique usability issues. Each usability issue was independently scored for severity by trained hospitalists and a Human Factors researcher and for safety severity by two physician informaticists and two clinical safety professionals.

Results Results demonstrated a positive correlation between heuristic severity and safety severity ratings. Regression analysis demonstrated that 49% of safety risk variability by clinical safety professionals (r = 0.70; n = 28) and 42% of safety risk variability by clinical informatics specialists (r = 0.65; n = 28) was explained by usability severity scoring of problems outlined by heuristic evaluation. Higher severity ratings of the usability issues were associated with increased perceptions of patient safety risk.

Discussion This study demonstrated the use of heuristic evaluation as a technique to quickly identify usability problems in an EHR that could lead to safety issues. Detection of higher severity ratings could help prioritize failures in EHR design that more urgently require design changes. This approach is a cost-effective technique for improving usability while impacting patient safety.

Conclusion Results from this study demonstrate the efficacy of the heuristic evaluation technique to identify usability problems that impact safety of the EHR. Also, the use of interdisciplinary teams for evaluation should be considered for severity assessment.

Keywords

user–computer interface - electronic health records - mobile application - heuristics - safety

Full Text

References

References
1 Adler-Milstein J, DesRoches CM, Kralovec P. , et al. Electronic health record adoption in US hospitals: progress continues, but challenges persist. Health Aff (Millwood) 2015; 34 (12) 2174-2180
2 Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med 2010; 363 (06) 501-504
3 Chaudhry B, Wang J, Wu S. , et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144 (10) 742-752
4 Kruse CS, Kristof C, Jones B, Mitchell E, Martinez A. Barriers to electronic health record adoption: a systematic literature review. J Med Syst 2016; 40 (12) 252
5 The Office of the National Coordinator for Health Information Technology. Test Procedure for §170.314(g)(3) Safety-enhanced design. Available at: https://www.healthit.gov/sites/default/files/standards-certification/2014-edition-draft-test-procedures/170-314-g-3-safety-enhanced-design-2014-test-procedures-draft-v-1.0.pdf
6 Couture B, Lilley E, Chang F. , et al. Applying user-centered design methods to the development of an mHealth application for use in the hospital setting by patients and care partners. Appl Clin Inform 2018; 9 (02) 302-312
7 Preece J, Rogers Y, Sharp H. Interaction Design: Beyond Human-Computer Interaction. 4th ed. Chichester: Wiley; 2015
8 Sorbello A, Ripple A, Tonning J. , et al. Harnessing scientific literature reports for pharmacovigilance. Prototype software analytical tool development and usability testing. Appl Clin Inform 2017; 8 (01) 291-305
9 Tarrell A, Grabenbauer L, McClay J. , et al. Toward improved heuristic evaluation of EHRs. Health Syst (Basingstoke) 2015; 4: 138-150
10 Nielsen J, Molich R. Heuristic evaluation of user interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Seattle, Washington, United States, April 1–0, 1990, 249–256. Available at: http://dl.acm.org/citation.cfm?id=97281 . Accessed February 10, 2017
11 Hertzum M, Jacobsen NE. The evaluator effect: a chilling fact about usability evaluation methods. Int J Hum Comput Interact 2003; 15: 183-204
12 Jeffries R, Desurvire H. Usability testing vs. heuristic evaluation. ACM SIGCHI Bull 1992; 24: 39-41
13 Molich R, Nielsen J. Improving a human-computer dialogue. Commun ACM 1990; 33: 338-348
14 Zhang J, Patel VL, Johnson TR. , et al. Evaluating and predicting patient safety for medical devices with integral information technology. In: Henriksen K, Battles JB, Marks ES. , et al., ed. Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology). Rockville, MD: Agency for Healthcare Research and Quality; 2005
15 Zhang J, Johnson TR, Patel VL, Paige DL, Kubose T. Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform 2003; 36 (1-2): 23-30
16 Cooper JB, Newbower RS, Long CD, McPeek B. Preventable anesthesia mishaps: a study of human factors. Anesthesiology 1978; 49 (06) 399-406
17 Graham MJ, Kubose TK, Jordan D, Zhang J, Johnson TR, Patel VL. Heuristic evaluation of infusion pumps: implications for patient safety in Intensive Care Units. Int J Med Inform 2004; 73 (11-12): 771-779
18 Allen M, Currie LM, Bakken S, Patel VL, Cimino JJ. Heuristic evaluation of paper-based Web pages: a simplified inspection usability methodology. J Biomed Inform 2006; 39 (04) 412-423
19 Tang Z, Johnson TR, Tindall RD, Zhang J. Applying heuristic evaluation to improve the usability of a telemedicine system. Telemed J E Health 2006; 12 (01) 24-34
20 Bates DW, Leape LL, Cullen DJ. , et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998; 280 (15) 1311-1316
21 Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003; 163 (12) 1409-1416
22 Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA 2018; 319 (12) 1276-1278
23 Walker JM, Carayon P, Leveson N. , et al. EHR safety: the way forward to safe and effective systems. J Am Med Inform Assoc 2008; 15 (03) 272-277
24 Han YY, Carcillo JA, Venkataraman ST. , et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116 (06) 1506-1512
25 Graber ML, Byrne C, Johnston D. The impact of electronic health records on diagnosis. Diagnosis (Berl) 2017; 4 (04) 211-223
26 Sittig DF, Wright A, Ash J. , et al. New unintended adverse consequences of electronic health records. IMIA Yearb 2016; (01) 7-12
27 Sittig DF, Ash JS, Singh H. Overview of safer guides. SAFER Electron Health Rec Saf Assur Factors EHR Resil 2015; 20: 153-156
28 Sittig DF, Singh H. Toward more proactive approaches to safety in the electronic health record era. Jt Comm J Qual Patient Saf 2017; 43 (10) 540-547
29 Lowry S, Quinn M, Ramaiah M. , et al. (NISTIR 7804) Technical Evaluation, Testing and Validation of the Usability of Electronic Health Records. NIST Interagency/Internal Report (NISTIR) - 7804. 2012. Available at: https://www.nist.gov/node/592206 . Accessed February 10, 2017
30 Borycki E, Kushniruk A, Carvalho C. A methodology for validating safety heuristics using clinical simulations: identifying and preventing possible technology-induced errors related to using health information systems. Comput Math Methods Med 2013. Doi: 10.1155/2013/526419
31 Nielsen J. Severity Ratings for Usability Problems. Nielsen Norman Group. 1995. Available at: https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/ . Accessed April 2, 2018
32 Ellsworth MA, Dziadzko M, O'Horo JC, Farrell AM, Zhang J, Herasevich V. An appraisal of published usability evaluations of electronic health records via systematic review. J Am Med Inform Assoc 2017; 24 (01) 218-226