Download PDF
Appl Clin Inform 2019; 10(02): 210-218
DOI: 10.1055/s-0039-1681073
Research Article
Georg Thieme Verlag KG Stuttgart · New York

Safeuristics! Do Heuristic Evaluation Violation Severity Ratings Correlate with Patient Safety Severity Ratings for a Native Electronic Health Record Mobile Application?

Brandan Kennedy
1   Medical Informatics and Telemedicine, Children's Mercy Hospital, Kansas City, Missouri, United States
2   Pediatric Hospital Medicine, Children's Mercy Hospital, Kansas City, Missouri, United States
3   Human Factors Collaborative, Children’s Mercy Hospital, Kansas City, Missouri, United States
,
Ellen Kerns
4   Department of Pediatrics, University of Nebraska Medical Center, Omaha, Nebraska, United States
,
Y. Raymond Chan
2   Pediatric Hospital Medicine, Children's Mercy Hospital, Kansas City, Missouri, United States
3   Human Factors Collaborative, Children’s Mercy Hospital, Kansas City, Missouri, United States
,
Barbara S. Chaparro
5   Human Factors and Behavioral Neurobiology, Embry-Riddle Aeronautical University, Daytona Beach, Florida, United States
,
Sarah D. Fouquet
3   Human Factors Collaborative, Children’s Mercy Hospital, Kansas City, Missouri, United States
› Author Affiliations
Further Information

Publication History

26 July 2018

31 January 2019

Publication Date:
27 March 2019 (online)

Also available at
   Permissions and Reprints

Abstract

Objective Usability of electronic health records (EHRs) remains challenging, and poor EHR design has patient safety implications. Heuristic evaluation detects usability issues that can be classified by severity. The National Institute of Standards and Technology provides a safety scale for EHR usability. Our objectives were to investigate the relationship between heuristic severity ratings and safety scale ratings in an effort to analyze EHR safety.

Materials and Methods Heuristic evaluation was conducted on seven common mobile EHR tasks, revealing 58 heuristic violations and 28 unique usability issues. Each usability issue was independently scored for severity by trained hospitalists and a Human Factors researcher and for safety severity by two physician informaticists and two clinical safety professionals.

Results Results demonstrated a positive correlation between heuristic severity and safety severity ratings. Regression analysis demonstrated that 49% of safety risk variability by clinical safety professionals (r = 0.70; n = 28) and 42% of safety risk variability by clinical informatics specialists (r = 0.65; n = 28) was explained by usability severity scoring of problems outlined by heuristic evaluation. Higher severity ratings of the usability issues were associated with increased perceptions of patient safety risk.

Discussion This study demonstrated the use of heuristic evaluation as a technique to quickly identify usability problems in an EHR that could lead to safety issues. Detection of higher severity ratings could help prioritize failures in EHR design that more urgently require design changes. This approach is a cost-effective technique for improving usability while impacting patient safety.

Conclusion Results from this study demonstrate the efficacy of the heuristic evaluation technique to identify usability problems that impact safety of the EHR. Also, the use of interdisciplinary teams for evaluation should be considered for severity assessment.

Keywords

user–computer interface - electronic health records - mobile application - heuristics - safety
 

  • References

  • 1 Adler-Milstein J, DesRoches CM, Kralovec P. , et al. Electronic health record adoption in US hospitals: progress continues, but challenges persist. Health Aff (Millwood) 2015; 34 (12) 2174-2180
    CrossrefPubMedSearch in Google Scholar
  • 2 Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med 2010; 363 (06) 501-504
    CrossrefPubMedSearch in Google Scholar
  • 3 Chaudhry B, Wang J, Wu S. , et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144 (10) 742-752
    CrossrefPubMedSearch in Google Scholar
  • 4 Kruse CS, Kristof C, Jones B, Mitchell E, Martinez A. Barriers to electronic health record adoption: a systematic literature review. J Med Syst 2016; 40 (12) 252
    CrossrefPubMedSearch in Google Scholar
  • 5 The Office of the National Coordinator for Health Information Technology. Test Procedure for §170.314(g)(3) Safety-enhanced design. Available at: https://www.healthit.gov/sites/default/files/standards-certification/2014-edition-draft-test-procedures/170-314-g-3-safety-enhanced-design-2014-test-procedures-draft-v-1.0.pdf
    PubMed
  • 6 Couture B, Lilley E, Chang F. , et al. Applying user-centered design methods to the development of an mHealth application for use in the hospital setting by patients and care partners. Appl Clin Inform 2018; 9 (02) 302-312
    Thieme ConnectPubMedSearch in Google Scholar
  • 7 Preece J, Rogers Y, Sharp H. Interaction Design: Beyond Human-Computer Interaction. 4th ed. Chichester: Wiley; 2015
    Search in Google Scholar
  • 8 Sorbello A, Ripple A, Tonning J. , et al. Harnessing scientific literature reports for pharmacovigilance. Prototype software analytical tool development and usability testing. Appl Clin Inform 2017; 8 (01) 291-305
    Thieme ConnectPubMedSearch in Google Scholar
  • 9 Tarrell A, Grabenbauer L, McClay J. , et al. Toward improved heuristic evaluation of EHRs. Health Syst (Basingstoke) 2015; 4: 138-150
    CrossrefPubMedSearch in Google Scholar
  • 10 Nielsen J, Molich R. Heuristic evaluation of user interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Seattle, Washington, United States, April 1–0, 1990, 249–256. Available at: http://dl.acm.org/citation.cfm?id=97281 . Accessed February 10, 2017
    PubMed
  • 11 Hertzum M, Jacobsen NE. The evaluator effect: a chilling fact about usability evaluation methods. Int J Hum Comput Interact 2003; 15: 183-204
    CrossrefPubMedSearch in Google Scholar
  • 12 Jeffries R, Desurvire H. Usability testing vs. heuristic evaluation. ACM SIGCHI Bull 1992; 24: 39-41
    PubMedSearch in Google Scholar
  • 13 Molich R, Nielsen J. Improving a human-computer dialogue. Commun ACM 1990; 33: 338-348
    CrossrefPubMedSearch in Google Scholar
  • 14 Zhang J, Patel VL, Johnson TR. , et al. Evaluating and predicting patient safety for medical devices with integral information technology. In: Henriksen K, Battles JB, Marks ES. , et al., ed. Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology). Rockville, MD: Agency for Healthcare Research and Quality; 2005
    Search in Google Scholar
  • 15 Zhang J, Johnson TR, Patel VL, Paige DL, Kubose T. Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform 2003; 36 (1-2): 23-30
    CrossrefPubMedSearch in Google Scholar
  • 16 Cooper JB, Newbower RS, Long CD, McPeek B. Preventable anesthesia mishaps: a study of human factors. Anesthesiology 1978; 49 (06) 399-406
    CrossrefPubMedSearch in Google Scholar
  • 17 Graham MJ, Kubose TK, Jordan D, Zhang J, Johnson TR, Patel VL. Heuristic evaluation of infusion pumps: implications for patient safety in Intensive Care Units. Int J Med Inform 2004; 73 (11-12): 771-779
    CrossrefPubMedSearch in Google Scholar
  • 18 Allen M, Currie LM, Bakken S, Patel VL, Cimino JJ. Heuristic evaluation of paper-based Web pages: a simplified inspection usability methodology. J Biomed Inform 2006; 39 (04) 412-423
    CrossrefPubMedSearch in Google Scholar
  • 19 Tang Z, Johnson TR, Tindall RD, Zhang J. Applying heuristic evaluation to improve the usability of a telemedicine system. Telemed J E Health 2006; 12 (01) 24-34
    CrossrefPubMedSearch in Google Scholar
  • 20 Bates DW, Leape LL, Cullen DJ. , et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998; 280 (15) 1311-1316
    CrossrefPubMedSearch in Google Scholar
  • 21 Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003; 163 (12) 1409-1416
    CrossrefPubMedSearch in Google Scholar
  • 22 Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA 2018; 319 (12) 1276-1278
    CrossrefPubMedSearch in Google Scholar
  • 23 Walker JM, Carayon P, Leveson N. , et al. EHR safety: the way forward to safe and effective systems. J Am Med Inform Assoc 2008; 15 (03) 272-277
    CrossrefPubMedSearch in Google Scholar
  • 24 Han YY, Carcillo JA, Venkataraman ST. , et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116 (06) 1506-1512
    CrossrefPubMedSearch in Google Scholar
  • 25 Graber ML, Byrne C, Johnston D. The impact of electronic health records on diagnosis. Diagnosis (Berl) 2017; 4 (04) 211-223
    CrossrefPubMedSearch in Google Scholar
  • 26 Sittig DF, Wright A, Ash J. , et al. New unintended adverse consequences of electronic health records. IMIA Yearb 2016; (01) 7-12
    PubMedSearch in Google Scholar
  • 27 Sittig DF, Ash JS, Singh H. Overview of safer guides. SAFER Electron Health Rec Saf Assur Factors EHR Resil 2015; 20: 153-156
    PubMedSearch in Google Scholar
  • 28 Sittig DF, Singh H. Toward more proactive approaches to safety in the electronic health record era. Jt Comm J Qual Patient Saf 2017; 43 (10) 540-547
    PubMedSearch in Google Scholar
  • 29 Lowry S, Quinn M, Ramaiah M. , et al. (NISTIR 7804) Technical Evaluation, Testing and Validation of the Usability of Electronic Health Records. NIST Interagency/Internal Report (NISTIR) - 7804. 2012. Available at: https://www.nist.gov/node/592206 . Accessed February 10, 2017
    PubMed
  • 30 Borycki E, Kushniruk A, Carvalho C. A methodology for validating safety heuristics using clinical simulations: identifying and preventing possible technology-induced errors related to using health information systems. Comput Math Methods Med 2013. Doi: 10.1155/2013/526419
    PubMed
  • 31 Nielsen J. Severity Ratings for Usability Problems. Nielsen Norman Group. 1995. Available at: https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/ . Accessed April 2, 2018
    PubMed
  • 32 Ellsworth MA, Dziadzko M, O'Horo JC, Farrell AM, Zhang J, Herasevich V. An appraisal of published usability evaluations of electronic health records via systematic review. J Am Med Inform Assoc 2017; 24 (01) 218-226
    CrossrefPubMedSearch in Google Scholar