Measuring Electronic Health Record Use in the Pediatric ICU Using Audit-Logs and Screen Recordings

Abstract

Background Time spent in the electronic health record (EHR) has been identified as an important unit of measure for health care provider clinical activity. The lack of validation of audit-log based inpatient EHR time may have resulted in underuse of this data in studies focusing on inpatient patient outcomes, provider efficiency, provider satisfaction, etc. This has also led to a dearth of clinically relevant EHR usage metrics consistent with inpatient provider clinical activity.

Objective The aim of our study was to validate audit-log based EHR times using observed EHR-times extracted from screen recordings of EHR usage in the inpatient setting.

Methods This study was conducted in a 36-bed pediatric intensive care unit (PICU) at Lucile Packard Children's Hospital Stanford between June 11 and July 14, 2020. Attending physicians, fellow physicians, hospitalists, and advanced practice providers with ≥0.5 full-time equivalent (FTE) for the prior four consecutive weeks and at least one EHR session recording were included in the study. Citrix session recording player was used to retrospectively review EHR session recordings that were captured as the provider interacted with the EHR.

Results EHR use patterns varied by provider type. Audit-log based total EHR time correlated strongly with both observed total EHR time (r = 0.98, p < 0.001) and observed active EHR time (r = 0.95, p < 0.001). Each minute of audit-log based total EHR time corresponded to 0.95 (0.87–1.02) minutes of observed total EHR time and 0.75 (0.67–0.83) minutes of observed active EHR time. Results were similar when stratified by provider role.

Conclusion Our study found inpatient audit-log based EHR time to correlate strongly with observed EHR time among pediatric critical care providers. These findings support the use of audit-log based EHR-time as a surrogate measure for inpatient provider EHR use, providing an opportunity for researchers and other stakeholders to leverage EHR audit-log data in measuring clinical activity and tracking outcomes of workflow improvement efforts longitudinally and across provider groups.

Keywords

EHRs and systems - EHR time - pediatrics - audit-log - databases - data validation and verification - intensive and critical care - workflow - physician

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by Stanford Institutional Review Board.

Publication History

Received: 29 January 2021

Accepted: 28 June 2021

Article published online:
11 August 2021

• References

• 1 Adler-Milstein J, Jha AK. HITECH act drove large gains in hospital electronic health record adoption. Health Aff (Millwood) 2017; 36 (08) 1416-1422
• 2 Washington V, DeSalvo K, Mostashari F, Blumenthal D. The HITECH era and the path forward. N Engl J Med 2017; 377 (10) 904-906
• 3 Office of the National Coordinator for Health Information Technology. “Office-based physician electronic health record adoption,” health IT quick-stat #50. Accessed January 2019 at: dashboard.healthit.gov/quickstats/pages/physician-ehr-adoption-trends.php
• 4 Office of the National Coordinator for Health Information Technology. “Percent of hospitals, by type, that possess certified health IT,” health IT quick-stat #52. dashboard.healthit.gov/quickstats/pages/certified-electronic-health-record-technology-in-hospitals.php . Accessed September 2018
• 5 Standards for Health Information Technology to Protect Electronic Health Information Created, Maintained, and Exchange =d, 45 C.F.R. § 170.210. Accessed 2021 at: https://www.law.cornell.edu/cfr/text/45/170.2102015
• 6 ASTM E2147-Standard Specification for Audit and Disclosure Logs for Use in Health Information Systems. Available at: https://www.astm.org/Standards/ E2147.htm
• 7 Sinsky CA, Rule A, Cohen G. et al. Metrics for assessing physician activity using electronic health record log data. J Am Med Inform Assoc 2020; 27 (04) 639-643
• 8 Rule A, Chiang MF, Hribar MR. Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods. J Am Med Inform Assoc 2020; 27 (03) 480-490
• 9 Adler-Milstein J, Adelman JS, Tai-Seale M, Patel VL, Dymek C. EHR audit logs: A new goldmine for health services research?. J Biomed Inform 2020; 101: 103343
• 10 Gardner RL, Cooper E, Haskell J. et al. Physician stress and burnout: the impact of health information technology. J Am Med Inform Assoc 2019; 26 (02) 106-114
• 11 DiAngi YT, Stevens LA, Halpern-Felsher B, Pageler NM, Lee TC. Electronic health record (EHR) training program identifies a new tool to quantify the EHR time burden and improves providers' perceived control over their workload in the EHR. JAMIA Open 2019; 2 (02) 222-230
• 12 Arndt BG, Beasley JW, Watkinson MD. et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017; 15 (05) 419-426
• 13 Hribar MR, Read-Brown S, Goldstein IH. et al. Secondary use of electronic health record data for clinical workflow analysis. J Am Med Inform Assoc 2018; 25 (01) 40-46
• 14 Read-Brown S, Hribar MR, Reznick LG. et al. Time requirements for electronic health record use in an academic ophthalmology center. JAMA Ophthalmol 2017; 135 (11) 1250-1257
• 15 Hribar MR, Read-Brown S, Reznick L. et al. Secondary use of EHR timestamp data: validation and application for workflow optimization. AMIA Annu Symp Proc 2015; 2015: 1909-1917
• 16 Tai-Seale M, Olson CW, Li J. et al. Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine. Health Aff (Millwood) 2017; 36 (04) 655-662
• 17 Hribar MR, Read-Brown S, Reznick L. et al. Secondary use of EHR timestamp data: validation and application for workflow optimization. AMIA Annu Symp Proc.. Accessed 2015 at: https://pubmed.ncbi.nlm.nih.gov/26958290/
• 18 Hribar MR, Huang AE, Goldstein IH. et al. Data-driven scheduling for improving patient efficiency in ophthalmology clinics. Ophthalmology 2019; 126 (03) 347-354
• 19 Wang JK, Ouyang D, Hom J, Chi J, Chen JH. Characterizing electronic health record usage patterns of inpatient medicine residents using event log data. PLoS ONE. 2019; 14 (02) e0205379
• 20 Baxter SL, Apathy NC, Cross DA, Sinsky C, Hribar MR. Measures of electronic health record use in outpatient settings across vendors. J Am Med Inform Assoc 2020
• 21 Unertl KM, Novak LL, Johnson KB, Lorenzi NM. Traversing the many paths of workflow research: developing a conceptual framework of workflow terminology through a systematic literature review. J Am Med Inform Assoc 2010; 17 (03) 265-273
• 22 Zheng K, Guo MH, Hanauer DA. Using the time and motion method to study clinical work processes and workflow: methodological inconsistencies and a call for standardized research. J Am Med Inform Assoc 2011; 18 (05) 704-710
• 23 Lopetegui M, Yen P-Y, Lai A, Jeffries J, Embi P, Payne P. Time motion studies in healthcare: what are we talking about?. J Biomed Inform 2014; 49: 292-299
• 24 Patel VL, Kannampallil TG, Kaufman DR. Cognitive Informatics for Biomedicine: Human Computer Interaction in Healthcare. Springe =r International Publishing; 2015.
• 25 National Research Network | Department of Medicine. Accessed May 28, 2020 at: https://medicine.ucsf.edu/center-clinical-informatics-and-improvement-research/national-research-network