Electronic Health Record Alert-Related Workload as a Predictor of Burnout in Primary Care Providers

Summary

Background: Electronic health records (EHRs) have been shown to increase physician workload. One EHR feature that contributes to increased workload is asynchronous alerts (also known as inbox notifications) related to test results, referral responses, medication refill requests, and messages from physicians and other health care professionals. This alert-related workload results in negative cognitive outcomes, but its effect on affective outcomes, such as burnout, has been under-studied.

Objectives: To examine EHR alert-related workload (both objective and subjective) as a predictor of burnout in primary care providers (PCPs), in order to ultimately inform interventions aimed at reducing burnout due to alert workload.

Methods: A cross-sectional questionnaire and focus group of 16 PCPs at a large medical center in the southern United States.

Results: Subjective, but not objective, alert workload was related to two of the three dimensions of burnout, including physical fatigue (p = 0.02) and cognitive weariness (p = 0.04), when controlling for organizational tenure. To reduce alert workload and subsequent burnout, participants indicated a desire to have protected time for alert management, fewer unnecessary alerts, and improvements to the EHR system.

Conclusions: Burnout associated with alert workload may be in part due to subjective differences at an individual level, and not solely a function of the objective work environment. This suggests the need for both individual and organizational-level interventions to improve alert workload and subsequent burnout. Additional research should confirm these findings in larger, more representative samples.

Citation: Gregory ME, Russo E, Singh H. Electronic health record alert-related workload as a predictor of burnout in primary care providers. Appl Clin Inform 2017; 8: 686–697 https://doi.org/10.4338/ACI-2017-01-RA-0003

Keywords

Electronic health records - workload - burnout - health information technology - safety

1. Background and Significance

Research on the rapid and widespread implementation of Electronic Health Records (EHRs) over the last several years has shed light on how these tools can improve efficiency but also increase work-related tasks for providers [[1]]. EHRs generally use two types of alerts, “synchronous” (i.e., alerts that pop up immediately in response to provider actions in the EHR), or “asynchronous” (i.e., notification type alerts sent to providers by others involved in patient care, usually in an “inbox-like” format [[2]], including information on test results, referrals, and patient or order status messages). Asynchronous alerts thus aim to improve communication, but our research shows that attending to these alerts contributes to a high workload for providers [[2],[3]]. For example, the average provider receives over 56 alerts and spends 49 minutes responding to asynchronous alerts per day [[2]], with many providers receiving little to no protected time to do so [[4]]. As a result, 86% of providers stated that they stay after hours and/or come in to work on weekends to respond to alerts [[4]].

Negative cognitive consequences of workload associated with asynchronous alerts (henceforth referred to as “alerts”) include information overload [[5]] and missed test results [[6]]. Yet, there has been a dearth of research on the relationship between alert workload and negative affective outcomes for providers. Of particular interest is burnout, defined as physical, cognitive, emotional, and interpersonal exhaustion due to chronic work stress [[7],[8]]. Burnout is an important affective outcome to study, as it has been found to be an outcome of high workload [[9]–[12]], and is associated with serious negative outcomes for organizations, patients, and providers. Prior work has suggested that burnout develops gradually over time, as one’s organizational tenure (defined as the length of time one has worked at an organization), increases [[13]–[15]]; however, levels of burnout vary between individuals due to individual, job, and work factors [[10]]. At the organizational level, burnout leads to costly turnover, which amounts to $250,000 or greater for each physician that must be recruited and replaced [[16]]. At the patient level, burnout is associated with dissatisfaction [[17]], and poor patient safety [[18]]. For providers, burnout has been associated with negative mental health outcomes, including depression and suicidal ideation [[19], [20]]. Unfortunately, burnout has been found to be more prevalent and more severe among physicians relative to the general population [[21]]. Further, among health care providers, burnout is particularly problematic in primary care, with meta-analytic evidence suggesting that primary care physicians have significantly higher burnout scores than specialists and other types of physicians [[22]]. We sought to examine the association between EHR alert-related workload and provider burnout.

2. Objectives

Our primary study goal was to examine asynchronous alert-related workload in the EHR as a predictor of burnout in primary care providers (PCPs), with the purpose of informing interventions targeted at reducing burnout associated with alert workload. We used both quantitative and qualitative methods to take a deep dive as a first step in establishing a relationship between alert workload and burnout, and to begin to understand what factors contribute to burnout.

Further, to better identify opportunities for interventions, we sought to distinguish between the contributions of objective and subjective alert workload to burnout. Objective workload is defined as the number of hours spent on work and work activities [[23]]. Subjective workload, in contrast, does not factor in actual time spent on work, but instead involves perceptions of having insufficient time to complete work tasks [[23]]. Counterintuitively, previous research has found little to no correlation between objective and subjective measures of workload [[24], [25]]. In addition, there appears to be a differential relationship between subjective and objective workload on affective work outcomes, with subjective workload predicting negative affect and objective workload predicting positive affect [[23]]. However, the extent to which objective and subjective alert workload contribute to burnout is largely unknown. In order to better tailor interventions to reduce workload-associated burnout, we must first understand whether to target objective workload (e.g., reduce the number of hours a task requires, or increase the allotted time to complete a task) or subjective workload (e.g., target psychological constructs that contribute to perceptions of high task load). Thus, as a subsidiary goal, we also explored the extent to which objective and subjective workload were predictive of burnout. This study was approved by the Baylor College of Medicine Institutional Review Board and the local VA Research Office.

3. Methods

3.1 Participants

Participants included a convenience sample of 16 PCPs (including physicians, nurse practitioners, and physicians’ assistants) at a large Veterans Affairs hospital in the southern United States. While we were unable to obtain precise data on the number of physicians, nurse practitioners, and physicians’ assistants in our sample, the composition of the unit is 73.7% physicians, 5.3% nurse practitioners, and 21.1% physicians’ assistants.

3.2 Procedure

At the beginning of regularly-scheduled staff meetings, we provided participants with a self-report questionnaire asking about organizational tenure, perceived alert burden (subjective alert workload), time spent responding to inbox-related alerts (objective alert workload), and burnout. We then engaged participants in hour-long semi-structured group discussions regarding attitudes towards alerts, facilitated by an experienced internal medicine physician from another department. Discussions took place over the course of four weeks, for a total of four hours’ worth of discussion. Discussions were not recorded; however, a researcher took notes to capture themes.

The EHR used by the PCPs in this study is the Veterans’ Affairs (VA) Computerized Patient Record System (CPRS). CPRS is the VA’s flagship EHR and is used by >150 VA facilities.

3.3 Measures

3.3.1 Subjective alert workload

We used a shortened, 5-item version of a self-report measure of perceived alert burden developed by Singh et al. [[4]]. Responses were captured on a 5-point Likert scale, where 1 = strongly disagree and 5 = strongly agree. The shortened scale was reliable, α = 0.74. Sample items include, “I receive too many alerts to easily focus on the most important ones,” and “The number of alerts exceeds what I can effectively manage.”

3.3.2 Objective alert workload

We asked providers how many hours they spent each day, on average, attending to alerts. If a range was provided, we took the average.

3.3.3 Burnout

We used a shortened and adapted 9-item version of the Shirom-Melamed Burnout Measure (SMBM) [[26]–[28]], which contains three subscales of burnout, each containing three items: physical fatigue (α = 0.95; sample item: “I feel like my ‘batteries’ are ‘dead’”), cognitive weariness (α = 0.96; sample item: “I feel I’m not thinking clearly”), and emotional exhaustion (α = 0.93; sample item: “I feel I am not capable of investing emotionally in coworkers and patients”). Responses were captured on a 7-point Likert scale, where 1 = never or almost never and 7 = always or almost always.

3.3.4 Organizational tenure

We asked providers how many months they had been employed at the organization.

3.4 Analysis

3.4.1 Quantitative

We conducted three multiple regression analyses, including one for each dimension of burnout (physical fatigue, cognitive weariness, and emotional exhaustion). The predictors were subjective alert workload and objective alert workload. We used organizational tenure as a covariate, as prior literature has suggested that burnout develops gradually over time [[13]–[15]]. Unless otherwise indicated, all tests are two-tailed. We considered p-values < 0.05 to be statistically significant; where applicable, we report lower p-values as they occur.

3.4.2 Qualitative

We took notes to capture participants’ responses to the semi-structured discussion questions. Additionally, we examined participants’ comments on an open-ended survey item: “Please provide any additional comments or feedback you have.” We analyzed responses to uncover themes regarding alert workload and burnout.

4. Results

4.1 Quantitative Data

►[Table 1] presents descriptive characteristics of our sample. As can be seen, PCPs had an average organizational tenure of 58 months (approximately six years), but there was a wide range, from one month to ten years. Participants were, on average, moderate to high on the alert burden and burnout scales, and spent an average of over two hours per day reviewing alerts.

Table 1

Descriptive statistics of sample (n = 16).

Variable	M	SD	Range
Organizational tenure, # months	58.50	43.39	1.00 – 121.00
Subjective alert workload (alert burden)[1]	3.78	0.83	2.20 – 4.80
Objective alert workload (average # hours spent reviewing alerts/day)	2.29	1.13	0.66 – 5.00
Burnout: Physical fatigue[2]	4.78	1.50	2.00 – 7.00
Burnout: Cognitive weariness[2]	3.07	1.53	1.00 – 5.67
Burnout: Emotional exhaustion[2]	2.98	1.53	1.00 – 5.67

Note. Four participants did not provide demographic information. M = Mean; SD = Standard deviation; 1 Scale ranges from 1–5, with 1 being the lowest score and 5 being the highest; 2 Scale ranges from 1–7, where 1 indicates low burnout and 7 indicates high burnout.

►[Table 2] shows correlations among all study variables. Subjective alert workload was strongly correlated with the physical fatigue dimension of burnout, whereas objective alert workload was not significantly correlated with any study variables. Notably, we found that objective and subjective workload were not significantly correlated with one another (r = 0.24, p > 0.05; ►[Table 2]).

Table 2

-Correlations among study variables.

Variable	1	2	3	4	5
1. Organizational tenure
2. Objective alert workload	−0.05
3. Burnout: Physical fatigue	−0.17	0.44
4. Burnout: Cognitive weariness	−0.37	0.06	0.64[**]
5. Burnout: Emotional exhaustion	−0.31	−0.01	0.59[**]	0.67[**]
6. Subjective alert workload	0.03	0.24	0.65[**]	0.37	0.21

** Note. p < 0.001

We conducted multiple regressions to examine the impact of subjective and objective alert workload on each of the three dimensions of burnout, while controlling for organizational tenure. For the physical fatigue dimension, we found that the overall model was significant (F = 3.76, p = .04, R²adj = 0.37), and that subjective alert workload was a significant predictor (β = 0.58, p = 0.02), but objective alert workload was not (β = 0.29, p = 0.21; ►[Table 3] and ►[Figure 1]). For the cognitive weariness dimension, the overall model was marginally significant (F = 3.20, p = 0.07, R²adj = 0.32), and subjective alert workload was a significant predictor (β = 0.53 p = 0.04), but objective alert workload was not (β = 0.00. p = 0.99; ►[Table 3] and ►[Figure 1]). The overall model for the emotional exhaustion dimension was not significant (F = 1.02, p =0.42, R²adj = 0.01), nor were subjective alert workload or objective alert workload significant predictors (β = 0.36, p = 0.23 and β = −0.04, p = 0.87, respectively; ►[Table 3] and ►[Figure 1]).

Table 3

Multiple regressions examining the relationship between alert workload and burnout dimensions.

	B	SE B	β	t	p
DV: Physical fatigue
Organizational tenure	−0.01	0.01	−0.22	−1.00	0.34
Objective alert workload	0.39	0.30	0.29	1.31	0.21
Subjective alert workload	1.14	0.45	0.58	2.56	0.02
DV: Cognitive weariness
Organizational tenure	−0.02	0.01	−0.58	−2.53	0.02
Objective alert workload	0.00	0.31	0.00	0.01	0.99
Subjective alert workload	1.02	0.46	0.53	2.23	0.04
DV: Emotional exhaustion
Organizational tenure	−0.02	0.01	−0.41	−1.47	0.17
Objective alert workload	−0.06	0.39	−0.04	−0.16	0.87
Subjective alert workload	0.74	0.59	0.36	1.27	0.23

Note. DV = Dependent variable; B = unstandardized estimate; SE = Standard error; β = standardized estimate, t = t value; p = p-value.

Fig. 1 Multiple regression of subjective and objective alert workload on the three dimensions of burnout, controlling of organizational tenure. Note: Numbers represent standardized beta-weights. *p < 0.05

In sum, we found that providers’ subjective alert workload was positively related to two of the three dimensions of burnout. We also found that subjective alert workload was generally predictive of burnout, whereas objective alert workload was not. These findings suggest that it is not the actual amount of time spent attending to EHR inbox alerts that predicts burnout, but rather, the perception of alert burden.

4.2 Qualitative Data

To further inform potential interventions, we thematically analyzed participants’ comments both on open-ended survey items and during the discussion. Two major themes regarding alert workload were uncovered, as described below.

4.2.1 Desire for protected time

Contrary to our quantitative findings, time spent managing alerts was a major theme in the focus group discussion and in survey comments. However, for many, it was not just the amount of time spent on alert management that contributed to burnout or the number of alerts received; rather, our participants indicated that they desired protected time for alert management. Participants currently do not have adequate protected time for this activity, and many find this challenging and frustrating. One specific quote included: “The # of alerts for me is less of an issue than the lack of protected administrative time that I am given to address the alerts, make follow up calls, and discuss results with my patients.”

This theme is of interest given our quantitative findings. It appears that PCPs perceive time spent managing asynchronous alerts (i.e., objective workload) to be a burden; however, our quantitative results show that this is not predictive of burnout. Regardless, allowing for protected time to respond to alerts may improve patient safety outcomes, such as fewer missed test results, by reducing the need to multi-task [[5]], and would, at a minimum, likely decrease perceptions of burden due to time spent managing alerts, and reported subsequent burnout.

4.2.2 Desire for improved EHR features

A second major theme of our qualitative data centered around the shortcomings of the EHR system that were perceived to induce a higher workload in regard to alert management. Specifically, to reduce workload, PCPs desired color-coded alerts, such that new alerts could be a different color than older alerts. In addition, PCPs wished for workflow enhancing features, such as easy auto-population of test results into templated letters that could be sent to patients, and for better access to their patients’ records from other facilities. PCPs also desired additional pieces of information to enhance the system, including how many alerts they had and time-stamps on messages from other providers.

Addressing these issues would likely improve both objective alert workload, by reducing cognitive overload, as well as subjective alert workload, by decreasing perceptions of alert burden. This could be tested initially by conducting an experimental, simulated, interventional study wherein these features are introduced into the EHR system and pre-post change in subjective and objective alert workload could be examined. However, we found many individual differences among provider preferences on how to decrease burden of alerts through better EHR design. For example, some participants actively sought to customize the alerts they receive (e.g., turning off optional alerts), and desired the aforementioned EHR tools to reduce the number of alerts and cognitive workload imposed by them. Conversely, participants on the opposite end of the spectrum valued every alert and did not wish to reduce the number of alerts they received. The latter participants cited reasons such as fear of missing important information or mistrust of the automated abnormal lab value ranges as reasons for wanting to review every alert.

5. Discussion

We examined the relationship between workload related to EHR inbox alerts and burnout using both quantitative and qualitative methods. Our findings suggest that subjective, but not objective, alert-related workload is related to two of the three dimensions of burnout. These findings are aligned with prior research showing that perceived ease of use of EHRs, but not number of alerts received, is associated with overlooking alerts of test results (i.e., alert fatigue) [[5]]. Collectively, this evidence suggests that subjective alert workload is likely an important predictor of negative outcomes.

Our quantitative findings are somewhat in contrast with a prior study that linked both subjective and objective workload to burnout with providers in the intensive care unit, although this study was not specific to alert-related workload [[29]]. Our qualitative results also somewhat contrast with our quantitative findings in that participants appear to be concerned with objective alert workload, as they desire protected time to respond to alerts, whereas our quantitative findings suggest time spent managing alerts was not predictive of burnout. This finding could be interpreted in terms of the job demands-job resources model [[30]], wherein perhaps protected time is perceived to be a potential resource for providers to cope with the demand of copious alerts.

Participants also indicated a desire for an improved EHR system, which could potentially reduce both objective and subjective alert workload and associated burnout. However, our qualitative results also suggest that individual differences cannot be ignored in future work or any intervention meant to reduce alert workload-related burnout, as there was a wide variety of views on alert burden, even within our small sample.

5.1 Implications

Altogether, these findings suggest that burnout associated with alert workload may be in part due to subjective differences at an individual level, and not solely a function of the objective work environment. This suggests that individual provider-level interventions should be developed and implemented. Characteristics of individuals more prone to burnout include largely malleable features, such as an external locus of control, and passive (as opposed to active) coping mechanisms [[11]]. Thus, to decrease burnout, training programs targeted at these characteristics (e.g., training for development of an external locus of control, training to utilize active coping mechanisms), should be developed and offered to providers. Additionally, to reduce perceptions of alert workload, training could also focus on strategies to manage EHR inbox alerts more efficiently [[31]]. A national initiative in the VA is now focusing on the latter type of training.

Prior research has also shown an association between use of other EHR components (such as computerized order entry systems), and burnout in providers, with these effects attributed to dissatisfaction with the amount of “administrative” time spent on clerical tasks [[32]]. Because several EHR components create additional work, there is a concomitant need to also provide improvements to the objective work environment. Such fixes can (and should) also be pursued not only to reduce workplace stressors [[33]], but also for safety reasons. Alerts can lead to information overload that can contribute to providers overlooking important test results, which could result in patient harm [[5]]. Organizational-level interventions could include reducing the number of alerts that a health care organization deems mandatory [[34]], streamlining alert delivery systems so that providers do not receive unnecessary or repetitive alerts [[34]], improving the design of the EHR [[35]], and devising strategies aimed to improve communication between PCPs and other non-PCP providers (for example, to prevent non-PCP providers from sending unnecessary alerts to PCPs) [[6], [34]]. Further, allowing protected time for providers to respond to alerts may simultaneously reduce workload and also improve patient safety by reducing the provider’s need to multitask. Nevertheless, it is important to consider these implications in context of our study’s limitations, which we describe below.

5.2 Limitations

There are a number of limitations in the current study. First, we acknowledge that our results are based on cross-sectional data. Thus, we were unable to examine the direction of our effects; yet, theory suggests that workload leads to burnout. In addition, we utilized a small convenience sample, which may limit our generalizability. We therefore caution readers that these findings should be confirmed using larger and more representative samples. We cannot be sure of the extent to which the results of our study apply to clinicians who have more resources to manage asynchronous alerts, such as those who already receive a large amount of protected time for alert management, or those who have nurses helping in managing most of the triage related to alerts. However, to the best of our knowledge, these situations are infrequent – in most systems, the largest chunk of alert burden is borne by the physician themselves, and most physicians do not get paid or get protected time for the additional time it takes to manage alerts. We are also unsure of the extent to which results apply to clinicians without EHR systems. Additionally, the self-report measure of objective workload may be biased or prone to recall error; however, prior work has shown no difference between self-report and observational methods on the reporting of time spent on indirect patient care tasks [[36]]. In fact, in our previous work on quantifying alerts [[4], [5]], the average self-reported number of alerts received was quite similar to the average we obtained via quantitative measurements using alert logs, suggesting that providers are able to estimate and report objective alert workload fairly accurately. Finally, as with any study, it is possible that there are confounders (e.g., factors unique to the study site), that may explain some of the results of this study. While we are unable to completely rule these out, our qualitative data support the idea that alert-related workload could, in fact, lead to burnout.

However, there are also some strengths in this study. First, the relationship between subjective workload and burnout held for two of the three dimensions of burnout, which partially mitigates concerns that this finding is spurious and a Type I error. Additionally, our sample was comprised of the actual end users of alerts on the EHR (i.e., PCPs), which allows for a deep understanding of the problem of objective vs. subjective alert workload from the perspective of PCPs (which is understudied), and subsequently increases external validity. Similarly, recent work using commercial EHRs suggests a similar workload burden outside this health care system [[3]], further supporting generalizability. Further, we used both quantitative and qualitative data to strengthen our findings.

5.3 Future Research

Future work should examine the association between individual differences in providers and their perceptions of alert workload. Additionally, institutions as well as providers should consider the usefulness of different strategies that might mitigate perceptions of alert workload. For instance, Singh [[34]] suggests that providers adjust the types of notifications they receive, take advantage of sorting and processing functionalities within the EHR to improve efficiency, and customize alerts to reduce duplicate notifications. Future research should evaluate the effectiveness of individual- and organizational-level interventions aimed at reducing alert workload and subsequent burnout. In addition, future work should test the relationships between subjective and objective alert workload and burnout in other samples. It is important to investigate these relationships in larger and more representative samples in order to obtain a more precise estimate of the effect sizes. For example, external validity could be improved by using a random sampling technique in the future, rather than convenience sampling. Stratified random sampling could be ideal, wherein the sampling frame accounts for differences in several key provider factors that theoretically may be or empirically have been demonstrated to be related to EHR use and experience [[37]], including provider type (physician, nurse practitioner, physician assistant), organizational tenure (novice vs. experienced), specialty (outpatient vs. inpatient; primary care vs. specialists), practice size (small vs. large), organization type (hospital vs. non-hospital practice), and hospital teaching classification (teaching vs. non-teaching).

6. Conclusions

A high subjective workload in regards to managing EHR inbox alerts was associated with PCP burnout, whereas objective alert workload was not a significant predictor. Participants desired having protected time for alert management, as well as improvements to the EHR system. These organizational-level interventions may reduce subjective alert workload and subsequent burnout. However, our findings suggest that individual differences should be taken into consideration when developing and implementing such interventions. Additionally, individual-level burnout interventions, such as coping-mechanism training, locus of control training, and training to improve the efficiency of inbox alert management, should be considered in tandem.

Multiple Choice Question

Which type of alert workload was predictive of burnout?

• Objective

• Subjective

• Organizational

• Perceptive

The correct answer is B). As stated in the paper, subjective alert workload (i.e., perceptions of burden associated with alerts) was predictive of two of the three dimensions of burnout (physical fatigue and cognitive weariness). Objective alert workload (i.e., the number of hours spent managing alerts) was not associated with any of the burnout dimensions. Organizational alert workload and perceptive alert workload are not real terms.

Conflict of Interest

The authors report no conflict of interest in this research.

Clinical Relevance Statement

While the knowledge on various components of workload and methods regarding how to measure these components is well-grounded [[23]–[25]], there is a dearth of studies that examine the impact of both objective and subjective workload on work outcomes. To our knowledge, this is one of the first to show that subjective, but not objective, workload leads to negative affective outcomes (i.e., burnout). This finding is important in guiding future research and in informing improvements in EHR design and processes to manage inbox alerts.

Human Subjects Protections

This study was approved by the Baylor College of Medicine Institutional Review Board and the local VA Research Office.

• References

• 1 Poissant L, Pereira J, Tamblyn R, Kawasumi Y. The impact of electronic health records on time efficiency of physicians and nurses: A systematic review. J Am Med Inform Assoc 2005; 12: 505-516.
• 2 Murphy DR, Reis B, Sittig DF, Singh H. Notifications received by primary care practitioners in electronic health records: A taxonomy and time analysis. Am J Med 2012; 125: 209-e1-e7.
• 3 Murphy DR, Meyer AN, Russo E, Sittig DF, Wei L, Singh H. The burden of inbox notifications in commercial electronic health records. JAMA Intern Med 2016; 176: 559-560.
• 4 Singh H, Spitzmueller C, Petersen NJ, Sawhney MK, Smith MW, Murphy DR, Espadas D, Laxmisan A, Sittig DF. Primary care practitioners’ views on test result management in EHR-enabled health systems: A national survey. J Am Med Inform Assoc 2012; 20: 727-735.
• 5 Singh H, Spitzmueller C, Petersen NJ, Sawhney MK, Sittig DF. (2013). Information overload and missed test results in electronic health record–based settings. JAMA Intern Med 2013; 173: 702-704.
• 6 Hysong SJ, Sawhney MK, Wilson L, Sittig DF, Esquivel A, Singh S, Singh H. Understanding the management of electronic test result notifications in the outpatient setting. BMC Med Inform Decis Mak 2011; 11: 1-9.
• 7 Dunford BB, Shipp AJ, Boss RW, Angermeier I, Boss AD. Is burnout static or dynamic? A career transition perspective of employee burnout trajectories. J Appl Psychol 2012; 97: 637-650.
• 8 Toker S, Biron M. Job burnout and depression: Unraveling their temporal relationship and considering the role of physical activity. J Appl Psychol 2012; 97: 699-710.
• 9 Aiken LH, Clarke SP, Sloane DM, Sochalski J, Silber JH. Hospital nurse staffing and patient mortality, nurse burnout, and job dissatisfaction. JAMA 2002; 288: 1987-1993.
• 10 Cordes CL, Dougherty TW. A review and an integration of research on job burnout. Acad Manage Rev 1993; 18: 621-656.
• 11 Maslach C, Schaufeli WB, Leiter MP. Job burnout. Annu Rev Psychol 2001; 52: 397-422.
• 12 McManus IC, Keeling A, Paice E. Stress, burnout and doctors’ attitudes to work are determined by personality and learning style: A twelve year longitudinal study of UK medical graduates. BMC Med 2004; 2: 1-12.
• 13 Brill PL. The need for an operational definition of burnout. Fam Community Health 1984; 6: 12-24.
• 14 Cherniss C. Staff burnout. Beverly Hills, CA: Sage; 1980
• 15 Hallsten L. Burning out: A framework. In: Schaufeli WB, Maslach C, Marek T. editors. Professional burn-out: Recent developments in theory and research. Philadelphia, PA:: Taylor & Francis; 1993. p. 95-113.
• 16 Buchbinder SB, Wilson M, Melick CF, Powe NR. Estimates of costs of primary care physician turnover. Am J Manag Care 1999; 5: 1431-1438.
• 17 Haas JS, Cook EF, Puopolo AL, Burstin HR, Cleary PD, Brennan TA. Is the professional satisfaction of general internists associated with patient satisfaction?. J Gen Intern Med 2000; 15: 122-128.
• 18 Hall LH, Johnson J, Watt I, Tsipa A, O’Connor DB. Healthcare staff wellbeing, burnout, and patient safety: A systematic review. PLoS One 2016; 11: e0159015.
• 19 Dyrbye LN, Thomas MR, Massie FS, Power DV, Eacker A, Harper W, Duming S, Moutier C, Szydlo DW, Novotny PJ, Sloan JA, Shanafelt TD. Burnout and suicidal ideation among US medical students. Ann Intern Med 2008; 149: 334-341.
• 20 Thomas NK. Resident burnout. JAMA 2004; 292: 2880-2889.
• 21 Shanafelt TD, Boone S, Tan L, Dyrbye LN, Sotile W, Satele D, West CP, Sloan J, Oreskovich MR. Burnout and satisfaction with work-life balance among US physicians relative to the general US population. Arch Intern Med 2012; 172: 1377-1385.
• 22 Mauthe AJ, Boudreau RA, Lalumiere ML. Physician burnout: A “meta-o-scopic” analysis. Poster presented at the 8^th International Conference on Occupational Stress and Health; Puerto Rico 2009
• 23 Van Emmerik IH, Jawahar IM. The independent relationships of objective and subjective workload with couples’ mood. Hum Relat 2006; 59: 1371-1392.
• 24 Hoonakker P, Carayon P, Gurses AP, Brown R, McGuire K, Khunlertkit A, Walker JM. (2011). Measuring workload of ICU nurses with a questionnaire survey: The NASA Task Load Index (TLX). IEE Trans Healthc Syst Eng 2011; 1: 131-143.
• 25 Matthews G, Reinerman-Jones LE, Barber DJ, Abich J. The psychometrics of mental workload multiple measures are sensitive but divergent. Hum Factors 2015; 57: 125-143.
• 26 Shirom A. Burnout in work organizations. In: Cooper CL, Robertson I. editors. International review of industrial and organization psychology. Chichester, U.K.:: Wiley; 1989: 25-48.
• 27 Shirom A. Job-related burnout: A review. In: Quick JC, Tetrick LE. editors. Handbook of occupational health psychology. Washington, DC:: American Psychological Association; 2003: 245-265.
• 28 Shirom A, Melamed S. A comparison of the construct validity of two burnout measures in two groups of professionals. Intl J Stress Manag 2006; 13: 176-200.
• 29 Schaufeli W, Le Blanc P. Personnel. In: Miranda DR, Ryan DW, Schaufeli WB, Fidler V. editors. Organisation and Management of Intensive Care: A Prospective Study in 12 European Countries. Berlin, Heidelberg:: Springer; 1998: 169-205.
• 30 Demerouti E, Bakker AB, Nachreiner F, Schaufeli WB. The job demands-resources model of burnout. J Appl Psychol 2001; 86: 499-512.
• 31 Singh H, Wilson L, Reis B, Sawhney MK, Espadas D, Sittig DF. Ten strategies to improve management of abnormal test result alerts in the electronic health record. J Patient Saf 2010; 6: 121-123.
• 32 Shanafelt TD, Dyrbye LN, Sinsky C, Hasan O, Satele D, Sloan J, West CP. Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction. Mayo Clin Proc 2016; 91: 836-848.
• 33 Sonnentag S, Frese M. Stress in organizations. In: Schmitt NW, Highhouse S. editors. Handbook of Psychology. 2^nd ed. Vol 12. Hoboken, NJ:: John Wiley & Sons, Inc.; 2013: 560-592.
• 34 Singh H. A checklist to improve CPRS ‘View Alert’ notifications: Recommendations for VA facility leadership and staff. Houston (TX):: Michael E. DeBakey VA Medical Center; 2016
• 35 Lowry SZ, Ramaiah M, Patterson ES, Brick D, Gurses AP, Ozok A, Simmons D, Gibbons MC. Integrating electronic health records into clinical workflow: An application of human factors modeling methods to ambulatory care. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care 2014; 3: 170-177.
• 36 Foster B, Williams RB. Substitution of self-reporting for observing time spent on work activities by mental health professionals. Psychol Rep 1989; 64: 945-946.
• 37 Simon SR, Kaushal R, Cleary PD, Jenter CA, Volk LA, Poon EG, Orav J, Lo HG, Williams DH, Bates DW. Correlates of electronic health record adoption in office practices: A statewide survey. J Am Med Inform Assoc 2007; 14: 110-117.