Electronic Health Record Use among Ophthalmology Residents while on Call

Keywords

electronic health records - audit logs - ophthalmology residents - call burden - clinical activity - EHR tasks - education policy

The rate of electronic health record (EHR) adoption now exceeds 80% among ophthalmologists nationwide.[1] Mandated in 2014 as part of the Health Insurance Portability and Accountability Act (HIPAA), EHR audit logs provide researchers with abundant, large-scale data on how users interact with their EHR system. These audit logs typically include time-stamped data of users' activities, including date, time, and location of log-ins, which patients' records were accessed, and what activities were performed. While originally intended for audit reasons to prevent inappropriate access of patient records, in recent years these logs have served as a rich source of data for understanding clinical workflows.[2] [3] [4] Previous studies have analyzed these data to characterize the activities of attending physicians, residents, and medical students alike.[5] [6] [7] [8]

Audit log data describe physicians' EHR use in granular detail and can be used to approximate work burden. Studies have shown that increased EHR use, particularly after-hours, is a key contributor to burnout for practicing physicians.[9] [10] [11] A 2015 nationwide study across all specialties documented that approximately 50% of physicians experience some form of burnout.[12] Burnout rates may be even higher among residents, and prior studies have shown ophthalmology residents to have especially high rates of burnout.[13] [14] [15] [16] [17] Although the Accreditation Council for Graduate Medical Education (ACGME) work hour restrictions implemented in 2003 and 2011 have been shown to decrease emotional fatigue and overall burnout rates, a significant number of residents across the country continue to report feelings of exhaustion and burnout.[18] With this in mind, previous work has characterized resident workflow and time allocation, particular after-hours while on call.[19] [20] [21] [22] [23] Although these studies span multiple specialties including internal medicine, otolaryngology, and vascular surgery, to our knowledge there have not been any published studies that characterize the activities of ophthalmology residents while on call outside of regular clinic hours. Furthermore, the majority of these prior studies relied on trained observers to record residents' actions, an approach that can be influenced by the Hawthorne effect (i.e., changing one's behavior due to the knowledge of being actively observed) and by variations in data collection and quality between different observers.

In this study, we aimed to objectively characterize the after-hours activities of ophthalmology residents while taking primary call using EHR audit log data. Examining patterns of EHR use among residents while on call may help inform education policies regarding call and influence strategies aimed at addressing physician burnout in the future.

Methods

Study Design and Population

This retrospective study was conducted at the University of California San Diego (UCSD). The study was approved by the UCSD Institutional Review Board and adhered to the tenets of the Declaration of Helsinki. Raw EHR audit logs were collected over a 12-month period from October 1, 2018, through September 30, 2019. October 2018 was designated as the start of the study period because the ophthalmology department implemented the enterprise-wide EHR (Epic Systems, Verona, WI) in late September 2018, after which all residents were asked to document all on-call interactions in the EHR. Previously, some on-call interactions (primarily those conducted with clinic patients) were documented on paper charts. However, the hospitals and emergency departments of the health system had implemented the EHR in 2010, and therefore even before the ophthalmology department implemented the EHR for ambulatory clinic encounters, ophthalmology residents had been accustomed to documenting on-call encounters from inpatient and emergency consults in the same system.

Per existing policies, first- and second-year ophthalmology residents were assigned to rotating primary call shifts, with third-year ophthalmology residents taking “back-up” call to help the primary call residents with any questions, typically answering questions over the phone and having limited interactions with patients and the EHR. Subsequently, for this study, we included data from only the first- and second-year residents taking primary call shifts, as they were the ones primarily interacting with patients and documenting in the EHR while on call. However, because the study period spanned portions of two academic cycles, a total of three classes of ophthalmology residents were included in the analysis. With four residents per class, this equated to 12 residents. PGY (postgraduate year) level was denoted by training year at the end of the study, as the study period spanned two academic year cycles.

Residents at this program also rotated and took call at the San Diego Veterans Affairs (VA) Healthcare System. However, consultation with the clinical informatics team revealed that granular EHR audit logs were not available at the time of this study. Therefore, EHR data from call activities at the VA could not be included.

EHR Audit Log Data Extraction and Analysis

Raw EHR audit logs were obtained for the eligible residents from the designated study period from the UCSD Clinical Data Warehouse. Logs were compared with resident schedules to isolate data gathered during resident call shifts. Data were analyzed based on the type of call shifts. Shifts were categorized as follows: weekday evenings (4 pm to midnight), weekday overnights (midnight to 8 a.m.), weekends (Saturdays and Sundays 8 a.m. to 8 a.m. the next day), and holidays (24-hour periods on select days throughout the year based on existing definitions by the university). Although in practice, on weekdays, a single resident would take primary call from 4 p.m. until the following morning at 8 a.m., for the purpose of the analysis, this was split into two portions (“evenings” and “overnights”) to characterize “middle-of-the-night” call activities (i.e., those after midnight) specifically, as these would be more disruptive to residents' quality of life and impose more burden through disruption of sleep cycles, which has been shown to be a key contributor to physician burnout.[24] [25] [26] Furthermore, a prior analysis in the internal medicine literature demonstrated substantial variations in clinical volumes and workload during nondaytime shifts, prompting the creation of a “swing shift” to accommodate the busiest hours of call between 4 p.m. and 11 p.m.[27] This provided further motivation to examine segments of weekday call shifts separately.

Data collected from EHR logs included login time, logout time, number of unique patient charts accessed, primary encounter diagnosis codes, and EHR tasks performed. Total time spent logged in per shift and the number of unique patient charts accessed (clinical volume) were calculated for each shift type. Due to variations in call shift length, time spent logged in the EHR and clinical volume were also standardized to per-hour metrics by dividing the EHR use time or clinical volume during the shift by the number of hours in the shift (8 hours for weekday evenings and weekday overnights, 24 hours for weekends and holidays). The most common diagnoses seen on call were tabulated using patients' primary encounter diagnosis codes listed in the EHR. Diagnoses were categorized into groups based on ophthalmic or general medicine/nonophthalmic diagnoses. All ophthalmic diagnoses were further grouped into categories such as general ophthalmology, plastics, pre-/postoperative, glaucoma, and so on. Of note, primary encounter diagnosis codes were generated by the primary provider for the overall encounter (i.e., admission diagnosis determined by an emergency physician or hospitalist) not the ophthalmology consultation specifically. Some diagnoses such as “null” and “other” were unable to be categorized and were excluded from the study.

Finally, to characterize residents' specific activities in the EHR, we extracted task descriptions provided by the EHR vendor that were recorded in the audit logs. Each specific task performed by a resident (such as viewing a note, opening a visit navigator, or looking up a specific part of a patient's chart) was recorded by the EHR system. We categorized these tasks into broader domains such as chart review, login/patient searching, documentation, and so on. Some metrics such as “masked data displayed,” “potential duplicates checked,” and “other” were unable to be categorized and were excluded from the study.

Statistical Analyses

For demographics and all outcomes of interest (EHR use time, clinical volume, diagnoses, EHR activities), descriptive statistics were generated using mean and standard deviation, median and interquartile range (IQR), or counts/proportions where appropriate. Linear models of EHR use time and clinical volume were used to evaluate differences between shift types. To evaluate trends in EHR use time and clinical volume longitudinally across the study period, data were aggregated and compared by monthly averages across all call shifts. Chi-square tests of goodness-of-fit were used to assess whether significant differences existed across months. To assess the relationship between clinical volume and EHR use time, we constructed a scatterplot between the two variables and fit a linear regression model to evaluate for a statistically significant correlation and to determine the R ² correlation coefficient. For all statistical analyses, significance was defined as p < 0.05. All analyses were performed using R version 3.5.1 (R Foundation for Statistical Computing, Vienna, Austria).[28]

Results

Participants and Shift Characteristics

EHR audit log data were collected for 12 ophthalmology residents taking primary call during the study period (Table 1). Based on self-report, the majority were female (8/12 [67%]). Eight (67%) were Asian and four (33%) were white. No residents self-reported identifying as Hispanic or Latino. Residents were evenly distributed by training year (four residents in each of three PGY training levels). Over the course of the year-long study, EHR data from 638 call shifts were obtained. Most call shifts corresponded to weekday/weekend designations as previously described. There were 15 (2%) holiday shifts based on existing definitions of university holidays.

Table 1 Demographics of ophthalmology residents (n = 12) whose electronic health record audit logs were analyzed from October 1, 2018, to September 30, 2019

Characteristics	N (% out of total n = 12)
Gender
Female	8 (67)
Male	4 (33)
Race
Asian	8 (67)
Caucasian	4 (33)
PGY level[a]
PGY-2	4 (33)
PGY-3	4 (33)
PGY-4	4 (33)

Abbreviation: PGY, postgraduate year.

^a PGY level denoted by training year at the end of the study (September 30, 2019), as the study period spanned two academic year cycles.

Electronic Health Record Use Time

Across all call shifts, the median (IQR) time spent logged into the EHR per shift was 88 (131) minutes. When standardized to per-hour measures, ophthalmology residents spent the most time using the EHR on weekday evening shifts (4 p.m. to midnight), with a median (IQR) of 12.1 (11.3) minutes per hour, followed closely by holidays (10.5 [5.4] minutes per hour; [Table 2]). Weekday evenings entailed significantly more EHR use time than weekends and weekday overnights (p < 0.001). Although weekday overnights between midnight and 8 a.m. entailed the least amount of EHR use for ophthalmology residents on a per-hour basis (4.75 [7.2]) compared with other types of call shifts, cumulatively there was still a broad range of EHR use during these middle-of-the-night hours, with median (IQR) of 38 (57.8) minutes across these shifts as a whole.

Table 2 Time and clinical volume associated with after-hours EHR use for ophthalmology residents for various call shifts over a 1-year period (October 1, 2018, to September 30, 2019)

Type of call shift	Median (IQR) number of minutes logged into the EHR			Median (IQR) number of unique patient charts accessed
	Across entire call shift	Standardized to per-hour basis[a]	p-Value[b]	Across entire call shift	Standardized to per-hour basis[a]	p-Value[b]
Weekday evenings (4 p.m. to midnight)	97 (90.0)	12.1 (11.3)	–	8.0 (6.0)	1.0 (0.8)	–
Weekday overnights (midnight to 8 a.m.)	38 (57.8)	4.75 (7.2)	<0.001	4.0 (3.0)	0.5 (0.4)	<0.001
Weekends (Saturdays and Sundays 8 a.m. to 8 a.m. the next day)	201 (213.5)	8.4 (8.9)	<0.001	14.5 (10.3)	0.6 (0.4)	<0.001
Holidays (24-h periods on select days)	253 (129.5)	10.5 (5.4)	0.23	17.0 (10.5)	0.7 (0.4)	0.014

Abbreviations: EHR, electronic health record; IR, interquartile range.

^a Values standardized to per-hour basis to allow cross-comparison across call shifts of different durations.

^b p-Values determined by linear models of the outcome with the type of call shift as the covariate. p-Values are based on differences between each shift type and weekday evenings (the reference shift type).

The total time spent logged in was greatest during the summer and fall months (August to December) and decreased during the spring months (downward trend between December and June; [Fig. 1A]). This seasonal variation did not reach statistical significance (p = 0.21).

Fig. 1 Longitudinal view of (A) time spent logged into the EHR per call shift and (B) number of unique patient charts accessed per call shift (October 2018 to September 2019). EHR, electronic health record.

Clinical Volume

The total number of patient charts accessed by on-call residents during the study period was 6,303. The median (IQR) unique patient charts accessed per shift was 7 (9) patients, regardless of type of call shift. Similar to the patterns of time spent logged into the EHR, on average ophthalmology residents encountered the highest clinical volumes during weekday evening call shifts, with median (IQR) of 1.0 (0.8) patients per hour followed by holidays (0.7 [0.4]; [Table 2]). The number of patient charts accessed during these shifts was significantly more than weekday overnights and weekends (p < 0.001). The number of unique patient charts accessed on call was also higher during the summer months compared with later parts of the academic year ([Fig. 1B]), although again this did not quite reach statistical significance (p = 0.16). Increased clinical volume was positively correlated with EHR use time ([Fig. 2]). Residents who accessed a greater number of patient charts during a call shift had a significantly longer duration of time logged into the EHR (p < 0.001; R ² = 0.51).

Fig. 2 Correlation between clinical volume (X-axis) and EHR use time (Y-axis). EHR, electronic health record.

Diagnoses and Electronic Health Record Activities

A total of 2,272 diagnoses were associated with the patients seen by ophthalmology residents on call. Of these, 1,611 (70.9%) were eye-related, 332 (14.6%) were postoperative, and 329 -(14.5%) were not eye-related. The top three diagnoses were “screening for eye condition” (285/2,272 [12.5%]), “postoperative state” (271/2,272 [11.9%), and dermatochalasis (110/2,272 [4.8%]). It is likely that patients carrying a diagnosis of “dermatochalasis” were actually status post blepharoplasty and were simply not characterized as “postoperative state” in the system, given that physicians often vary in assignment of diagnosis codes. Diagnoses were grouped based on field and further subdivided into categories, as seen in [Table 3]. The ophthalmic categories with the greatest number of diagnoses were plastics (427/2,272 [18.8%]), general ophthalmology/anterior segment (351/2,272 [15.4%]), and pre-/postoperative (332/2,272 [14.6%]).

Table 3 Categorization of primary encounter diagnosis codes for cases seen on call by ophthalmology residents

	Diagnosis category	Example diagnoses	Number (total N = 2,272)	%
Ophthalmic	Plastics	Dermatochalasis thyroid eye disease, eyelid lesion	427	18.8
	General/nonspecific	Screening for eye condition, visual disturbance, eye pain	351	15.4
	Glaucoma	Glaucoma suspect, primary open-angle glaucoma, congenital glaucoma	276	12.1
	Cataracts	Lens replaced, cataract, nuclear sclerotic cataract	162	7.1
	Retina/uveitis	Retinal detachment, uveitis, diabetic retinopathy	123	5.4
	Trauma	Assault, fall, trauma, ruptured globe	87	3.8
	Infectious/inflammatory	Orbital cellulitis, chalazion, preseptal cellulitis, corneal ulcer	81	3.6
	Cornea/refractive	Corneal abrasion, keratoconus, myopia	60	2.6
	Ocular surface issues	Dry eye syndrome, allergic conjunctivitis, conjunctivitis	44	1.9
Postoperative	Pre-/postoperative	Postoperative state, Preoperative evaluation, H/O eye surgery	332	14.6
Nonophthalmic	Cardiac/vascular	Aortic valve replacement, essential hypertension, hyperlipidemia	132	5.8
	Infectious	Sepsis, HIV disease, cellulitis	60	2.6
	Oncology	Skin neoplasm, chronic lymphocytic leukemia, metastatic melanoma	61	2.7
	Neuro/psychiatric	Headache, altered mental status, seizure	42	1.8
	General	Routine medical examination, diarrhea	34	1.5

Abbreviation: HIV, human immunodeficiency virus.

EHR activities were tabulated in a similar fashion. A total of 104,682 individual tasks were recorded in EHR audit logs for ophthalmology residents on call during the study period. The top three task descriptions (as listed by the EHR vendor) were “notes viewed” (14,079/104,682 [13.4%]), “report with patient data viewed” (13,549/104,682 [12.9%]), and “visit navigator template” (5,634/104,682 [5.4%]). We grouped tasks based on activity domains, as seen in [Table 4]. The most common categories of EHR activities were chart review (66,372/104,682 [63.4%]), login/patient searching (17,230/104,682, 16.5%), and documentation (11,719/104,682 [11.2%]).

Table 4 Categorization of ophthalmology resident tasks performed within the EHR system while on-call

Task category	Top examples	Number (total N = 104,682)	%
Chart review	Notes viewed, patient data report viewed, encounter viewed in chart review	66,372	63.4
Login, patient searching	User authenticated, patient selected from patient lookup, patient lookup search	17,230	16.5
Documentation	Visit navigator template, sign clinical note, eye exam saved, diagnosis updated	11,719	11.2
Inbox/communication	In basket message created, in basket message viewed, communication management accessed	4,962	4.7
Orders	Orders viewed, order entry accessed, open inpatient order set	3,876	3.7
Administrative	Patient data report printed, patient emergency contacts accessed, order printed from chart review	523	0.5

Abbreviation: EHR, electronic health record.

Discussion

Based on a broad-based literature search of several databases (PubMed, Embase, and Scopus), to our knowledge this study represents the first analysis of work patterns of ophthalmology residents while on call using EHR audit log data. Our key findings were as follows: (1) EHR use time and clinical volume varied based on the type of call shift and season, (2) chart review comprised a majority of ophthalmology residents' on-call EHR activities, and (3) EHR audit logs demonstrated substantial call burden for ophthalmology residents outside of regular clinic hours. Overall, EHR audit logs represent an emerging data source for improving understanding educational experience and work hours, and informing future policies and interventions to mitigate resident burnout. As detailed in this study, residency programs can potentially utilize these data to better understand their residents' training experience.

First, we found variations in EHR use time and clinical volume between different call shift types. When standardized to per-hour metrics, weekday evenings were the busiest call shifts, with a higher number of patients and longer EHR use time compared with other types of call shifts. Although weekends and holidays were both 24-hour shifts, holidays were busier based on EHR use time and on the number of patient charts. Program directors could potentially integrate these types of data from their individual institutions when formulating call schedules. Currently, call shifts are generally assigned to achieve an equal number of days to each resident within various categories (e.g., weeknights, weekends, holidays). However, there may be additional sources of variation, for example, certain holidays may be consistently busier than others. Historical data regarding EHR use time and patient volume could therefore refine future scheduling efforts. Additionally, similar to previous studies, the number of patients seen was highest during the summer months and lowest in the spring.[6] Although the differences did not quite reach statistical significance, there were clearly trends of seasonal variation. This likely reflects academic cycles, as new interns and residents start in the summer months and are likely to call more ophthalmology consults. Time spent logged in also followed a similar pattern. This may have resulted from an increased workload as described previously. However, for new PGY-2 residents, greater time spent logged in at the beginning of the academic year may have also reflected learning a new EHR system and learning how to evaluate ophthalmology patients for the first time. The log data alone were not sufficient to understand the differential effects of these two learning processes.

In terms of activities performed in the EHR, our study estimates that more than 60% of ophthalmology residents' EHR tasks on call were geared toward chart review. This supports prior studies across multiple specialties, which have reported a similar disproportionate amount of EHR activity geared toward medical chart review during regular work hours.[6] [7] [29] [30] Interestingly, this contrasts studies of daytime activities of ophthalmology attending physicians, where audit log analyses have shown that the number and percentage of notes reviewed were very low.[31] The authors of that study found that ancillary staff accessed significantly more notes than ophthalmologists. While they contended that ophthalmology attending physicians were not using the vast majority of content in the EHR during daytime outpatient encounters, here we found that ophthalmology residents are still dedicating the majority of their on-call EHR activity to chart review, likely by necessity as they are often encountering new patients with acute problems and are typically not supported by ancillary staff. This information is valuable to inform future EHR training sessions for ophthalmology residents, where more efficient chart search functions and navigation features could be emphasized. Thus, using EHR audit log data to characterize usage patterns can facilitate more targeted training for residents, who may have different needs than attending physicians. Additionally, these results could be used to improve patient data visualization and chart summation tools by identify common resident EHR activities. Prior studies have found that training quality has a significant impact on perceived efficiency and EHR satisfaction,[32] and therefore these strategies could improve residents' experience.

The data reflect the breadth of educational exposure of these residents, as well as substantial work activities after-hours while on call. Ophthalmology residents accessed patients with a wide range of diagnoses. Combining the median values for weekday evening (4 p.m. to midnight) and overnight (midnight to 8 a.m.), ophthalmology residents accessed a median of 12 patient charts on weekday calls between two regularly scheduled workdays. These patients are in addition to patients accessed during busy daytime rotations. To provide context, studies have estimated that internal medicine residents view an average of 14 unique charts per day.[29] Although a few ophthalmology residency programs have “eye emergency departments” with dedicated shifts (meaning the resident is “in house” during the call shift and then does not work the next day), the vast majority of ophthalmology residency programs have “home call” policies, where residents travel between clinical sites and their homes to see ophthalmology consultations and are often expected to work full days both before and after the call shift. Although weekday overnight shifts (midnight to 8 a.m.) were the least busy in terms of EHR use time and the number of patient charts accessed, ophthalmology residents still accessed a median of four charts during those middle-of-the-night hours, during which work tends to be the most disruptive to sleep–wake cycles. Weekends and holidays also imposed substantial work burden.

Our data have some limitations. They underestimated the actual time spent interacting with the EHR and the time the residents spent working on call for several reasons. First, one hospital in which residents take primary call in this program was excluded from the study. This hospital used a different EHR system from which comparable data could not be obtained. Furthermore, this study did not capture all residents' on-call activities, such as performing surgeries or procedures, patient encounters when the resident was interacting with patients but not actively logged into the EHR, or time spent on telephone calls speaking with patients. Lastly, we recognize there are several additional factors contributing to resident call experience, including length of call shift and time spent driving between clinical sites (not an insignificant burden given that the distance between the university eye clinic and the level 1 trauma center exceeded 13 miles). Therefore, the true work burden on call exceeds what is represented here, although the log data at least provide some representation of the minimum. In addition, this study was limited to analysis of EHR use time while residents were taking primary call and did not include after-hours EHR use during noncall periods. Given a prior study showing significant after-hours EHR use by ophthalmology attending physicians after regularly scheduled clinics,[33] it is possible that residents may have incurred similar after-hours EHR use on noncall days that was not represented here. However, our observations during prior time-motion studies of ophthalmology residents[34] demonstrated that in faculty clinics, residents generally completed documentation during the clinic prior to the faculty attending ophthalmologist initiating their interaction with the patient. However, in resident-run clinics where residents serve as primary providers rather than in supporting roles (such as at the VA), there may be after-hours burden of EHR documentation even when not on call. Finally, this was also a single-center analysis examining residents within a single program. In future studies, it will be interesting to compare variation in call burden and clinical exposure between different hospitals and regions of the country. Larger sample sizes of residents may also enable adequate power to evaluate differences in time expenditures and EHR tasks across different demographic groups based on gender, race, and/or ethnicity.

In summary, this study leverages EHR audit log data to describe ophthalmology residents' activities while on call. This is a more objective approach that can supplement resident self-report, which is the current standard. Using EHR audit log data for understanding residents' experiences may have broader implications in the fields of physician burnout and education policy. As evidenced previously, these methods can be used to roughly estimate work hours and educational exposure of resident physicians. There have been several studies focused on addressing these goals in the field of internal medicine,[35] [36] [37] but ongoing work is needed to apply these methods to surgical specialties such as ophthalmology, where programs are smaller and residents often take home call to cover multiple clinical sites. Other foreseeable applications of EHR audit log data could be to identify programs violating work-hour restrictions or even to better inform evolving work-hour policies in the future. This is particularly relevant because residents may underreport their work hours, particularly if after-hours EHR use is considered.[38] [39] Given the growing epidemic of physician burnout, understanding how to leverage these data to inform strategies for improving physicians' experience at a critical (and vulnerable) stage of their training is increasingly important.

• References

• 1 Lim MC, Boland MV, McCannel CA. , et al. Adoption of electronic health records and perceptions of financial and clinical outcomes among ophthalmologists in the United States. JAMA Ophthalmol 2018; 136 (02) 164-170
• 2 Hribar MR, Read-Brown S, Reznick L. , et al. Secondary use of EHR timestamp data: validation and application for workflow optimization. AMIA Annu Symp Proc 2015 2015; 1909-1917
• 3 Hribar MR, Biermann D, Read-Brown S. , et al. Clinic workflow simulations using secondary EHR data. AMIA Annu Symp Proc 2017 2016; 647-656
• 4 Hribar MR, Read-Brown S, Goldstein IH. , et al. Secondary use of electronic health record data for clinical workflow analysis. J Am Med Inform Assoc 2018; 25 (01) 40-46
• 5 Tai-Seale M, Olson CW, Li J. , et al. Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine. Health Aff (Millwood) 2017; 36 (04) 655-662
• 6 Chen L, Guo U, Illipparambil LC. , et al. Racing against the clock: internal medicine residents' time spent on electronic health records. J Grad Med Educ 2016; 8 (01) 39-44
• 7 Wang JK, Ouyang D, Hom J, Chi J, Chen JH. Characterizing electronic health record usage patterns of inpatient medicine residents using event log data. PLoS One 2019; 14 (02) e0205379
• 8 Chi J, Kugler J, Chu IM. , et al. Medical students and the electronic health record: ‘an epic use of time’. Am J Med 2014; 127 (09) 891-895
• 9 Arndt BG, Beasley JW, Watkinson MD. , et al. Tethered to the EHR: Primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017; 15 (05) 419-426
• 10 Tran B, Lenhart A, Ross R, Dorr DA. Burnout and EHR use among academic primary care physicians with varied clinical workloads. AMIA Jt Summits Transl Sci Proc 2019; 2019: 136-144
• 11 Downing NL, Bates DW, Longhurst CA. Physician burnout in the electronic health record era: are we ignoring the real cause?. Ann Intern Med 2018; 169 (01) 50-51
• 12 Shanafelt TD, Hasan O, Dyrbye LN. , et al. Changes in burnout and satisfaction with work-life balance in physicians and the general US working population between 2011 and 2014. Mayo Clin Proc 2015; 90 (12) 1600-1613
• 13 Dyrbye LN, Burke SE, Hardeman RR. , et al. Association of clinical specialty with symptoms of burnout and career choice regret among US resident physicians. JAMA 2018; 320 (11) 1114-1130
• 14 Alotaibi AK, Alsalim A, Alruwaili F. , et al. Burnout during ophthalmology residency training: a national survey in Saudi Arabia. Saudi J Ophthalmol 2019; 33 (02) 130-134
• 15 Collier R. Burnout symptoms common among medical residents but rates vary across specialties. CMAJ 2018; 190 (41) E1239
• 16 Tran EM, Scott IU, Clark MA, Greenberg PB. Resident wellness in US ophthalmic graduate medical education: the resident perspective. JAMA Ophthalmol 2018; 136 (06) 695-701
• 17 Tran EM, Scott IU, Clark MA, Greenberg PB. Assessing and promoting the wellness of United States ophthalmology residents: a survey of program directors. J Surg Educ 2018; 75 (01) 95-103
• 18 Busireddy KR, Miller JA, Ellison K, Ren V, Qayyum R, Panda M. Efficacy of interventions to reduce resident physician burnout: a systematic review. J Grad Med Educ 2017; 9 (03) 294-301
• 19 Chaiyachati KH, Shea JA, Asch DA. , et al. Assessment of inpatient time allocation among first-year internal medicine residents using time-motion observations. JAMA Intern Med 2019; 179 (06) 760-767
• 20 Fletcher KE, Visotcky AM, Slagle JM, Tarima S, Weinger MB, Schapira MM. The composition of intern work while on call. J Gen Intern Med 2012; 27 (11) 1432-1437
• 21 Lurie N, Rank B, Parenti C, Woolley T, Snoke W. How do house officers spend their nights? A time study of internal medicine house staff on call. N Engl J Med 1989; 320 (25) 1673-1677
• 22 Melzer J. Clouds of different colors: a prospective look at head and neck surgical resident call experience. Ear Nose Throat J 2017; 96 (12) E19-E24
• 23 Aziz F, Talhelm L, Keefer J, Krawiec C. Vascular surgery residents spend one fifth of their time on electronic health records after duty hours. J Vasc Surg 2019; 69 (05) 1574-1579
• 24 Kancherla BS, Upender R, Collen JF. , et al. Sleep, fatigue and burnout among physicians: an American Academy of Sleep Medicine position statement. J Clin Sleep Med 2020; 16 (05) 803-805
• 25 Kancherla BS, Upender R, Collen JF. , et al. What is the role of sleep in physician burnout?. J Clin Sleep Med 2020; 16 (05) 807-810
• 26 Stewart NH, Arora VM. The impact of sleep and circadian disorders on physician burnout. Chest 2019; 156 (05) 1022-1030
• 27 Al-Kofahi M, Mohyuddin GR, Taylor ME, Eck LM. Reducing resident physician workload to improve well being. Cureus 2019; 11 (06) e5039
• 28 RStudio Team. RStudio: Integrated Development for R. RStudio, PBC, Boston, MA.. 2020 . Available at: http://www.rstudio.com/
• 29 Ouyang D, Chen JH, Hom J, Chi J. Internal medicine resident computer usage: an electronic audit of an inpatient service. JAMA Intern Med 2016; 176 (02) 252-254
• 30 Cox ML, Farjat AE, Risoli TJ. , et al. Documenting or operating: where is time spent in general surgery residency?. J Surg Educ 2018; 75 (06) e97-e106
• 31 Hribar MR, Biermann D, Goldstein IH, Chiang MF. Clinical documentation in electronic health record systems: analysis of patient record review during outpatient ophthalmology visits. AMIA Annu Symp Proc 2018 2018; 584-591
• 32 Longhurst CA, Davis T, Maneker A. Arch Collaborative. , et al; Local investment in training drives electronic health record user satisfaction. Appl Clin Inform 2019; 10 (02) 331-335
• 33 Read-Brown S, Hribar MR, Reznick LG. , et al. Time requirements for electronic health record use in an academic ophthalmology center. JAMA Ophthalmol 2017; 135 (11) 1250-1257
• 34 Gali HE, Baxter SL, Lander L. , et al. Impact of electronic health record implementation on ophthalmology trainee time expenditures. J Acad Ophthalmol 2019; 11 (02) e65-e72
• 35 Shine D, Pearlman E, Watkins B. Measuring resident hours by tracking interactions with the computerized record. Am J Med 2010; 123 (03) 286-290
• 36 Ouyang D, Chen JH, Krishnan G, Hom J, Witteles R, Chi J. Patient outcomes when housestaff exceed 80 hours per week. Am J Med 2016; 129 (09) 993-999.e1
• 37 Dziorny AC, Orenstein EW, Lindell RB, Hames NA, Washington N, Desai B. Automatic detection of front-line clinician hospital shifts: a novel use of electronic health record timestamp data. Appl Clin Inform 2019; 10 (01) 28-37
• 38 Gilleland M, Komis K, Chawla S, Fernandez S, Fishman M, Adams M. Resident duty hours in the outpatient electronic health record era: inaccuracies and implications. J Grad Med Educ 2014; 6 (01) 151-154
• 39 Bennett CL, McDonald DA, Chang Y. , et al. A National cross-sectional study of surgery residents who underreport duty hours. J Surg Educ 2017; 74 (06) 928-933