Usability Evaluation of Electronic Health Record System around Clinical Notes Usage–An Ethnographic Study

Abstract

Background A substantial gap exists between current Electronic Health Record (EHR) usability and potential optimal usability. One of the fundamental reasons for this discrepancy is poor incorporation of a User-Centered Design (UCD) approach during the Graphical User Interface (GUI) development process.

Objective To evaluate usability strengths and weaknesses of two widely implemented EHR GUIs for critical clinical notes usage tasks.

Methods Twelve Internal Medicine resident physicians interacting with one of the two EHR systems (System-1 at Location-A and System-2 at Location-B) were observed by two usability evaluators employing an ethnographic approach. User comments and observer findings were analyzed for two critical tasks: (1) clinical notes entry and (2) related information-seeking tasks. Data were analyzed from two standpoints: (1) usability references categorized by usability evaluators as positive, negative, or equivocal and (2) usability impact of each feature measured through a 7-point severity rating scale. Findings were also validated by user responses to a post observation questionnaire.

Results For clinical notes entry, System-1 surpassed System-2 with more positive (26% vs. 12%) than negative (12% vs. 34%) usability references. Greatest impact features on EHR usability (severity score pertaining to each feature) for clinical notes entry were: autopopulation (6), screen options (5.5), communication (5), copy pasting (4.5), error prevention (4.5), edit ability (4), and dictation and transcription (3.5). Both systems performed equally well on information-seeking tasks and features with greatest impacts on EHR usability were navigation for notes (7) and others (e.g., looking for ancillary data; 5.5). Ethnographic observations were supported by follow-up questionnaire responses.

Conclusion This study provides usability-specific insights to inform future, improved, EHR interface that is better aligned with UCD approach.

Keywords

electronic health records - interfaces - usability - graphical user interface - clinical documentation - qualitative - methodology

Note

The content is solely the responsibility of the authors and does not represent the official views of the National Science Foundation or Agency for Healthcare Research and Quality.

Protection of Human and Animal Subjects

Residents interacting with two different EHR systems in their respective hospitals were investigated following approvals and in compliance with Institutional Review Board (IRB) no.1308E41121 and Research and Development committee (R&D) no. R140720X standards.

Funding

This work was supported by National Science Foundation Award no. CMMI-1150057 (J.L.M.) and the Agency for Healthcare Research and Quality Award no. R01HS022085 (G.B.M.).

• References

• 1 Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Aff (Millwood) 2011; 30 (03) 464-471
• 2 Zhang J, Walji MF. TURF: toward a unified framework of EHR usability. J Biomed Inform 2011; 44 (06) 1056-1067
• 3 Holden RJ. Physicians' beliefs about using EMR and CPOE: in pursuit of a contextualized understanding of health IT use behavior. Int J Med Inform 2010; 79 (02) 71-80
• 4 Kossman SP, Scheidenhelm SL. Nurses' perceptions of the impact of electronic health records on work and patient outcomes. Comput Inform Nurs 2008; 26 (02) 69-77
• 5 Rodriguez NJ, Murillo V, Borges JA, Ortiz J, Sands DZ. A usability study of physicians interaction with a paper-based patient record system and a graphical-based electronic patient record system. Proc AMIA Symp 2002; 667-671
• 6 Barnum CM. . Usability testing essentials ready, set…test! Burlington, Massachusetts, MA: Morgan Kaufmann; 2010
• 7 ISO. ISO 9241-11:1998 Ergonomic requirements for office work with visual display terminals (VDTs)–Part 11: guidance on usability. ISO 1998; 45: 22
• 8 Nielsen J. Usability Engineering. Boston: Academic Press; 1993
• 9 Nielsen J. Usability 101: Introduction to Usability 2003. Available at: http://www.nngroup.com/articles/usability-101-introduction-to-usability/
• 10 Baxter K. Understanding Your Users: A Practical Guide to User Research Methods. 2nd ed. Waltham, Massachusetts, MA: Morgan Kaufmann; 2015
• 11 Jones S, Donelle L. Assessment of electronic health record usability with undergraduate nursing students. Int J Nurs Educ Scholarsh 2011; 8 (01) 24
• 12 Friedberg MW, Chen PG, Van Busum KR. , et al. Factors affecting physician professional satisfaction and their implications for patient care, health systems, and health policy. Rand Health Q 2014; 3 (04) 1
• 13 Carayon P, Cartmill R, Blosky MA. , et al. ICU nurses' acceptance of electronic health records. J Am Med Inform Assoc 2011; 18 (06) 812-819
• 14 Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc 2015; 22 (06) 1179-1182
• 15 Beynon-Davies P. Ethnography and information systems development: ethnography of, for and within is development. Inf Softw Technol 1997; 39 (08) 531-540
• 16 Saleem JJ, Plew WR, Speir RC. , et al. Understanding barriers and facilitators to the use of clinical information systems for intensive care units and anesthesia record keeping: a rapid ethnography. Int J Med Inform 2015; 84 (07) 500-511
• 17 McMullen CK, Ash JS, Sittig DF. , et al. Rapid assessment of clinical information systems in the healthcare setting: an efficient method for time-pressed evaluation. Methods Inf Med 2011; 50 (04) 299-307
• 18 Kopp BJ, Erstad BL, Allen ME, Theodorou AA, Priestley G. Medication errors and adverse drug events in an intensive care unit: direct observation approach for detection. Crit Care Med 2006; 34 (02) 415-425
• 19 Middleton B, Bloomrosen M, Dente MA. , et al; American Medical Informatics Association. Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA. J Am Med Inform Assoc 2013; 20 (e1): e2-e8
• 20 Kaipio J, Lääveri T, Hyppönen H. , et al. Usability problems do not heal by themselves: National survey on physicians' experiences with EHRs in Finland. Int J Med Inform 2017; 97: 266-281
• 21 Zahabi M, Kaber DB, Swangnetr M. Usability and safety in electronic medical records interface design: a review of recent literature and guideline formulation. Hum Factors 2015; 57 (05) 805-834
• 22 Georgsson M, Staggers N. An evaluation of patients' experienced usability of a diabetes mHealth system using a multi-method approach. J Biomed Inform 2016; 59: 115-129
• 23 Khajouei R, Hasman A, Jaspers MW. Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system. Int J Med Inform 2011; 80 (05) 341-350
• 24 Walji MF, Kalenderian E, Piotrowski M. , et al. Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR. Int J Med Inform 2014; 83 (05) 361-367
• 25 Walji MF, Kalenderian E, Tran D. , et al. Detection and characterization of usability problems in structured data entry interfaces in dentistry. Int J Med Inform 2013; 82 (02) 128-138
• 26 Devine EB, Lee CJ, Overby CL. , et al. Usability evaluation of pharmacogenomics clinical decision support aids and clinical knowledge resources in a computerized provider order entry system: a mixed methods approach. Int J Med Inform 2014; 83 (07) 473-483
• 27 Clarke MA, Belden JL, Kim MS. Determining differences in user performance between expert and novice primary care doctors when using an electronic health record (EHR). J Eval Clin Pract 2014; 20 (06) 1153-1161
• 28 Ant Ozok A, Wu H, Garrido M, Pronovost PJ, Gurses AP. Usability and perceived usefulness of Personal Health Records for preventive health care: a case study focusing on patients' and primary care providers' perspectives. Appl Ergon 2014; 45 (03) 613-628
• 29 Corrao NJ, Robinson AG, Swiernik MA, Naeim A. Importance of testing for usability when selecting and implementing an electronic health or medical record system. J Oncol Pract 2010; 6 (03) 120-124
• 30 Rizvi RF, Harder KA, Hultman GM. , et al. A comparative observational study of inpatient clinical note-entry and reading/retrieval styles adopted by physicians. Int J Med Inform 2016; 90: 1-11
• 31 Rosenbloom ST, Denny JC, Xu H, Lorenzi N, Stead WW, Johnson KB. Data from clinical notes: a perspective on the tension between structure and flexible documentation. J Am Med Inform Assoc 2011; 18 (02) 181-186
• 32 Mamykina L, Vawdrey DK, Stetson PD, Zheng K, Hripcsak G. Clinical documentation: composition or synthesis?. J Am Med Inform Assoc 2012; 19 (06) 1025-1031
• 33 Brown PJ, Marquard JL, Amster B. , et al. What do physicians read (and ignore) in electronic progress notes?. Appl Clin Inform 2014; 5 (02) 430-444
• 34 Reeves S, Kuper A, Hodges BD. Qualitative research methodologies: ethnography. BMJ 2008; 337: a1020
• 35 Berg BL, Lune H, Lune H. Qualitative research methods for the social sciences. Boston, Massachusetts, MA: Pearson; 2004
• 36 Timestamped field notes. 2015 ; Available at: http://www.neukadye.com/mobile-applications/timestamped-field-notes/
• 37 Chen L, Guo U, Illipparambil LC. , et al. Racing against the clock: internal medicine residents' time spent on electronic health records. J Grad Med Educ 2016; 8 (01) 39-44
• 38 Altheide DL. Reflections: ethnographic content analysis. Qual Sociol 1987; 10 (01) 65-77
• 39 Srnka KJ, Koeszegi ST. From words to numbers: how to transform qualitative data into meaningful quantitative results. Schmalenbach Business Review 2007; 59 (01) 29-57
• 40 NVivo-QSR International. 2015 ; Available at: http://www.qsrinternational.com/products_nvivo.aspx