Keywords pneumonia - implementation - clinical decision support - continuous quality improvement - health informatics - medical decision-making
Background and Significance
Background and Significance
As the leading infectious cause of death in the United States, pneumonia is a major target for quality improvement. Timely and accurate decision-making surrounding diagnosis, site of care, and treatment with antibiotics is crucial to optimize outcomes and typically occurs through a complex integration of information from the patient and electronic health record (EHR) ([Fig. 1 ]). Adherence to best-practice guidelines for the management of pneumonia has been associated with improved outcomes.[1 ]
[2 ]
[3 ] Local adaptation and implementation of best-practice guidelines is thus a grade I recommendation.[4 ]
[5 ] However, adherence to guidelines is low, and widespread variation in practice and outcomes exists,[6 ]
[7 ]
[8 ]
[9 ]
[10 ] which may be due to provider uncertainty in the guidelines.[11 ] The best approaches to adaptation of evidence-based practice across health care systems are not well defined, and differences in settings, patient populations, and providers may create contextual challenges to standardizing practice.[12 ]
Fig. 1 Workflow.
Computerized clinical decision support (CDS) embedded in the EHR is a promising way to implement best practices reliably and sustainably across a health care system,[13 ] although few CDS tools for pneumonia have successfully impacted practice or outcomes.[14 ]
[15 ]
[16 ] We implemented a computerized CDS for pneumonia, “ePneumonia,” across four emergency departments (EDs). In an ecological pre–post study design, we found that implementation of ePneumonia was associated with a reduction of 30-day mortality, increase in high-risk hospitalizations, and increase in first-line antibiotics use compared with three control EDs that used paper-based guidelines without the CDS (previously reported[17 ]). However, we found no reduction in hospitalizations of low-risk patients, which we had anticipated based upon other implementation efforts[18 ] and examination of baseline practice patterns.[19 ] Computerized CDS offers the opportunity to probe more deeply into the rejection or adoption of best-practice recommendations, generate important feedback for CDS improvement, and enable us to learn from and engage physicians. We thus sought to examine physician interactions with the CDS to identify factors driving rejection or adoption of CDS recommendations.
Objectives
The aims of this study were to:
Track the adoption versus rejection of best-practice guideline recommendations provided to ED physicians through their interaction with a bedside CDS.
Examine clinical reasons provided by physicians for rejecting best-practice recommendations.
Methods
Setting and Intervention
Intermountain Healthcare comprises 22 hospitals in Utah and Idaho and has been a pioneer in the development of a clinically oriented electronic health data and CDS.[20 ] In 1998, a paper-based pneumonia guideline was implemented with moderate success in all EDs.[21 ]
[22 ] In 2011, we implemented the computerized CDS tool at four of seven EDs in the urban central region of Salt Lake City for ED physicians caring for patients diagnosed with suspected pneumonia that integrates individual patient data with guideline-based recommendations at the point of care.
Description of CDS Design and Implementation Process
ePneumonia is integrated into the physician workflow at a critical point in decision-making, when the physician has typically completed an initial evaluation of the patient and is synthesizing the patient's history, physical examination, and results of laboratory and radiographic tests to form a diagnosis and treatment plan ([Fig. 1 ]). The tool initially screens all ED patients with chest imaging for evidence of pneumonia based upon a Bayesian analysis of clinical data, including natural language processing of chest imaging reports, vital signs, chief complaint, and laboratory values.[23 ] If the estimated likelihood of pneumonia exceeds 40%, the provider is alerted through the electronic ED patient tracking board, which displays continuously updated information on ED patient census and status displayed on every ED work station computer. Additionally, the physician can initiate ePneumonia independently of the screening tool through a desktop icon. If he/she confirms suspected pneumonia, ePneumonia proceeds to extract patient age and comorbidities, vital signs, nursing assessments, laboratory values, prior microbiology data, and radiographic evidence of pleural effusions and multilobar infiltrates. Using this information, ePneumonia automatically calculates a 30-day mortality risk estimate,[24 ] and identifies hospital admission criteria,[19 ] severe community-acquired pneumonia criteria,[25 ] and risk factors for infection with resistant organisms (“health care-associated pneumonia”).[4 ] The tool integrates the patient data, risk assessments, and guideline-based management recommendations, including site of care (intensive care unit [ICU], hospital ward, or outpatient), diagnostic studies, and antibiotic selection, onto four sequential screens.
All decisions and orders for patient care remain under the control and responsibility of the ED provider, who is fully supported regardless of whether he/she accepts or rejects any of the recommendations. When a recommendation is rejected, ePneumonia asks the physician to provide a reason. Common reasons for rejection of recommendations identified in the literature[6 ]
[26 ]
[27 ] are offered in a structured dropdown menu; however, a text box is also available for physicians to leave unstructured text as either an addition or alternative to the prespecified responses.
We collaborated with clinician stakeholders during the development of ePneumonia through outreach to the four participating EDs, reviewing standard management of pneumonia with ED physicians and meeting with clinical leadership. We conducted preliminary testing of the tool in one of the EDs during December 2010 to May 2011, including face-to-face visits with physicians during clinical shifts to receive feedback to improve the design. ePneumonia was implemented into routine clinical care at four hospital EDs in May 2011. A usability survey of 72 physicians with experience using the CDS was conducted in November 2011, in which physicians reported high satisfaction, specifically in its usefulness for antibiotic recommendations and site-of-care decisions ([Supplementary Material ], available in the online version).
Participants
We identified all cases in which providers used ePneumonia during the period of December 1, 2011 through November 30, 2012. Providers could use the tool multiple times for the same patient to refresh the clinical data; when this occurred, we examined results only from the final iteration. Cases in which the tool was launched but not completed to a treatment recommendation were excluded.
Measurements
For each provider–CDS interaction, we measured rejection or adoption of two major recommendations: (1) site of care (hospitalization to ICU or medical ward, or discharge from the ED), and (2) antibiotic selection. Physicians could proceed to the site-of-care recommendation but then stop before receiving the antibiotic selection recommendation; for these cases, we evaluated the site-of-care recommendations only. When the physician rejected the tool recommendation, we collected structured data of prespecified reasons from the provided dropdown menu selections, as well as free-text entries entered by the physician. Levels of reported satisfaction with the CDS were also measured among those physicians who completed the postimplementation usability survey ([Supplementary Material: Survey Methods and Results ], available in the online version).
Analysis
Comparisons between proportions of cases with agreements/disagreements were tested for significance using Fisher's exact and chi-square tests, where appropriate. We used generalized estimating equations analysis to examine the relationships between repeated measures of physician agreement (yes vs. no) with site-of-care and antibiotic recommendations and the following provider variables: age, gender, years worked, attending or resident, number of encounters, and usability of ePneumonia based on survey responses ([Supplementary Material ]), available in the online version).
We analyzed the unstructured text left by physicians rejecting recommendations by applying concept mapping, a mixed methods analysis that combines group sorting with multidimensional scaling.[28 ] Two clinician investigators (N.D. and B.J.) reviewed each entry, merged, and parsed the constructs to ensure similar length and level of abstraction. Then, 15 practicing clinicians including physicians and nurse practitioners independently participated in a Delphi-type card sorting exercise, in which those cards that represented a similar concept were sorted together. We applied multidimensional scaling to the card sorting data by computing average distances between concepts and generated cluster trees according to average distances based upon complete linkage. N.D. and B.J. then identified themes representative of each cluster.
The study was approved by the Intermountain Institutional Review Board (IRB #1017598). Implied consent was obtained from all surveyed physicians by completion of the survey and from clinicians participating in the card-sorting exercise by their participation, and both were approved by the IRB; waiver of patient consent for CDS data collection was approved by the IRB. All statistical analyses were performed using SPSS (Version 19.0. IBM SPSS Statistics for Windows, Armonk, New York, United States) or STATA MP (Version 14.1, StataCorp, College Station, Texas, United States); the card-sorting exercise and concept mapping analysis was performed using the X-Sort software (http://xsortapp.com ). Data analysis code is available upon request.
Results
The CDS provided site-of-care recommendations for 1,722 patient encounters, and antibiotic recommendations for 1,507 during the study period. Adoption of antibiotic recommendations was slightly higher at Intermountain Medical Center (95% vs. 89% at lowest-agreeing facility, p < 0.01); we found no significant between-hospital differences in adoption of site-of-care recommendations.
Physicians rejected site-of-care recommendations in 16% of visits, of which 84% reflected a provider's decision to place the patient in a higher-acuity site of care than recommended. Physicians rejected the outpatient recommendation in 24% of visits. Among the prespecified reasons for rejecting recommendations, physicians most commonly selected greater severity of illness than determined by the CDS, and uncontrolled comorbidities requiring hospitalization ([Table 1 ]). Providers rejecting site-of-care recommendations left free-text reasons for rejection in 53% of cases. Concept mapping and multidimensional scaling ([Fig. 2 ]) revealed two reasons for rejection of the outpatient recommendation that overlapped with prespecified constructs—severity of illness not extracted by the tool and clinical comorbidities—and three new themes: alternative or additional diagnoses, risks for outpatient failure due to functional status, and risk of outpatient failure due to social comorbidities such as homelessness and lack of access to care ([Table 1 ]).
Fig. 2 Revision. Concept mapping site-of-care deviations.
Table 1
Reasons for rejection of best-practice recommendations
Percent (N ) of disagreements
Site-of-care (N = 266, 16% of all cases)
Prespecified reasons for rejection
Patient sicker than estimated
56% (148)
Clinical judgment
22% (58)
Uncontrolled comorbid illnesses
17% (46)
Patient less sick than estimated
15% (43)
Patient preference
6% (17)
Oxygen requirements not appreciated by tool
5% (12)
No caregiver support
4% (10)
Critical care needs not appreciated by tool
3% (8)
Failed outpatient therapy
3% (8)
Can't tolerate PO meds
1% (3)
Pregnancy
0.4% (1)
Provider left free-text entry
53% (141)
Additional themes from provider free-text entries
Alternative/additional diagnoses
21% (58)
Risk of outpatient failure:
Functional status
4% (12)
Social comorbidities
2% (6)
Antibiotics (
N
= 104, 7% of all cases):
Prespecified reasons for rejection
Medication allergy
38% (39)
Immune compromised
10% (10)
Risk factors for MRSA not identified by tool
3% (3)
Risk factors for anaerobes not identified by tool
3% (3)
Provider left free-text entry
75% (78)
Additional themes from provider free-text entries
Previous treatments (antibiotics, interventions)
20% (20)
Preferences of admitting physician
11% (11)
Differences in pathogen risk assessment
9% (9)
Abbreviations: MRSA, methicillin-resistant Staphylococcus aureus ; PO, twice daily.
Physicians rejected antibiotic recommendations in only 7% of cases. Medication allergies not extracted by the CDS were the most common prespecified reasons for rejection ([Table 1 ]). The most common antibiotic recommendation rejected was to prescribe broad-spectrum antibiotics for patients meeting the criteria for health care-associated pneumonia, with a rejection rate of 10%; 76% provided free-text reasons for deviating. Concept mapping and multidimensional scaling ([Fig. 3 ]) also revealed properties of the antibiotics such as allergy potential. New themes identified included additional patient history including previous treatments, preferences of the admitting hospital physician, and differences in pathogen risk assessment ([Table 1 ]).
Fig. 3 Revision. Concept mapping antibiotic deviations.
Adoption of the site-of-care recommendation was positively associated with increased physician age, but negatively associated with perception of the tool's usefulness ([Table 2 ]). In contrast, adoption of the antibiotic recommendations was positively associated with younger physician age and perception of the tool's usefulness ([Table 3 ]). We found no significant relationships between adoption and other provider characteristics including gender, utilization, attending versus resident status, or other survey responses of satisfaction with the CDS.
Table 2
Physician characteristics and adoption of site-of-care recommendation
Odds ratio
Lower 95% CI
Upper 95% CI
p -Value
Age ≤ 40
1
Age 41–50
1.3
0.8
2.3
0.26
Age 51–60
2.7
1.3
5.5
0.007
Age > 60
4.1
1.5
5.5
0.007
Years worked in ED
0.95
0.75
0.99
0.009
Overall tool usefulness (1–5)
0.80
0.67
0.96
0.016
Usefulness for ordering diagnostic studies (1–5)
0.87
0.75
0.99
0.009
Experience of technical difficulties (1–5)
0.77
0.66
.91
0.002
Abbreviations: CI, confidence interval; ED, emergency department; GEE, generalized estimating equations.
Note: GEE regression model of N = 1,293 interactions and 58 physicians. Additional variables in the model found not to be significant included provider gender, number of tool uses per physician, reported usefulness of screening tool, and reported impact on clinical activity.
Table 3
Physician characteristics versus adoption of antibiotic recommendation
Odds ratio
Lower 95% CI
Upper 95% CI
p -Value
Age ≤ 40
1
Age 41–50
0.22
0.11
0.45
< 0.001
Age 51–60
0.30
0.11
0.84
0.02
Age > 60
0.26
0.10
0.65
0.004
Overall tool usefulness (1–5)
2.0
1.4
2.8
< 0.001
Abbreviations: CI, confidence interval; ED, emergency department; GEE, generalized estimating equations.
Note: GEE regression model of N = 1,293 interactions, 58 physicians. Additional variables in the model included provider gender, years worked in ED, reported usefulness of screening tool, usefulness for ordering diagnostic studies, and usability of tool.
Discussion
In a four-hospital ED implementation of a computerized CDS for pneumonia that improved care process, we found that adoption of recommendations was high, but physicians who rejected best-practice recommendations in practice provided new reasons for rejection not previously highlighted in the literature, including alternative diagnoses and additional patient information used for decision-making not extracted by the tool that led to uncertainty in the recommendations. Examining physician–CDS interactions was feasible and provided insights to the implementation and adaptation of evidence-based practice in pneumonia.
Effective implementation of innovations in a complex system requires the ability of users to constantly reinvent and adapt innovations to different contexts.[29 ] Effective adaptation of CDS can be supported through the participation by users at the individual patient level to generate continuous feedback.[30 ] CDS designs are never perfect at the time of implementation and require some form of surveillance, but system-level monitoring can be time-consuming and costly.[31 ] By studying the interaction between physicians and a computerized CDS for pneumonia during its implementation, we sought to develop a sustainable approach to understand the adaptation process for continuous learning and CDS improvement.
CDS tools have been increasingly proposed as a way to improve processes in acute health care settings.[32 ] The ED is an ideal setting where CDS may enhance decision-making, specifically by reducing cognitive errors that can occur under time pressure and information overload.[33 ] However, a recent review of computerized CDS in the ED concluded that few studies have demonstrated a positive impact.[34 ] A lack of high-quality intervention studies may explain this failure to observe effectiveness, but it may also be a function of CDS implementation, which can require substantial outreach and understanding of barriers to CDS utilization.[35 ] For pneumonia, implementation of guideline-based CDS may be further challenged by provider uncertainty in the applicability or appropriateness of those guidelines, or in the information used by CDS to generate individual recommendations. The importance of usability in CDS is widely recognized,[36 ]
[37 ] as is the potential for qualitative research to identify barriers to use of CDS.[38 ] In order for implementation of CDS to be successful, systems must be adapted locally, and they must be built to empower clinicians to provide feedback for continued development and learning.[39 ]
While we found a high rate of adoption in best-practice recommendations among physicians using our CDS, we also found that physicians who rejected recommendations were motivated to leave unstructured text entries in most cases. We had designed the CDS to integrate well into the workflow of a busy ED provider; thus, we took care to provide “quick” options from the dropdown menu. We were surprised by the number of physicians who contributed free-text entries given their time constraints, often giving information with themes that overlapped with those from the prespecified constructs. This resulted in a dialogue between the physician and the CDS and its creators, allowing the physician to teach the tool exceptions to its rules, and enabling us to refine our design. The dialogue may also heighten provider's awareness when objective data integrated by the tool is inconsistent with his/her assessment.
Some reasons for rejection were expected, and related to discordance in information used by physicians versus the CDS. Failure to identify allergies was the most common reason for rejection of the antibiotic recommendation, due to unreliable allergy information within the EHR. The EHR often either missed relevant antibiotic allergies, or more frequently provided incorrect information such as classifying nausea from amoxicillin as a penicillin allergy. Thus, providers had to manually revise allergy information at the time of the recommendation, leading to some discordance. We have since worked to improve the CDS' extraction of accurate allergy information.
We found that 84% of rejections of the site-of-care decision resulted in a higher acuity of care. This is consistent with our previously reported finding that the CDS increased high-risk hospitalizations, but did not significantly reduce low-risk hospitalizations. Failure to capture the full complexity and context of individual patients is a known limitation of both CDS and pneumonia guidelines, and our finding that physicians used additional information beyond the tool to assess severity was consistent with other studies.[12 ]
[40 ] Physicians consider a multitude of other factors when choosing site of care, and tools that predict 30-day mortality or ICU needs are not a perfect surrogate for the estimation the physician truly has to make: the risk of failure if a patient receives the counterfactual/opposite of the treatment under consideration (i.e., if the patient is discharged home). Social comorbidities such as homelessness and mental illness are crucial to decision-making but are difficult to glean from structured data. This is a known blind spot for computerized CDS. Applying more clinically relevant risk prediction, and developing approaches such as natural language processing of clinical documents to identify unstructured data such as social risk factors,[41 ]
[42 ] can improve CDS' ability to better align with physician decision-making and are the subjects of future work.
Other reasons for rejection represented new findings, which our thematic analysis of text revealed. We identified alternative diagnoses as an additional important reason to reject guideline recommendations not previously highlighted in the literature. Diagnostic uncertainty in pneumonia could be caused by many factors: several other diagnoses can present similarly, microbiologic confirmation is rarely identified, and accurate diagnosis can be challenged by time pressure and desire to treat.[43 ]
[44 ] The diagnosis of pneumonia often changes during a hospital course.[45 ] CDS may have an important role in supporting clinicians' uncertainty, by synthesizing disconfirming evidence, suggesting alternative diagnoses, or recommending additional diagnostic testing. This may better align with decision-making, help reduce anchoring, and enhance providers' ability to consider alternative diagnoses throughout the clinical course.
Our study has some limitations. The CDS required voluntary participation by providers; thus, our results represent feedback from only those physicians who used it. Our examination of provider characteristics suggested a relationship between provider age and guideline adherence for site-of-care and antibiotic use. However, additional provider and patient characteristics in a larger sample of physicians would better inform these relationships. Our focus was on capturing provider-reported reasons for deviating from guidelines, so we did not examine patterns of disagreements by patient characteristics, as previous studies have done.[12 ] Reported reasons for rejecting may not always reflect the “true” reasons that physicians might deviate. Individuals are often not aware of factors affecting decision-making such as cognitive load, uncertainty, bias, economic or time pressures, or practice norms.[46 ] Responses may also be influenced by social desirability.[47 ] Citing patient factors as reasons for rejecting guidelines may be a more socially desirable and conscious response than reporting personal uncertainty. Our future work is directed at examining both reported and unreported patient and provider characteristics that influence physician behavior.
Our results have stimulated ongoing, continuous improvements of ePneumonia in our system. Since our evaluation, we have incorporated additional clinical data that providers reported using to make decisions, including continuous updates of patient data throughout the encounter, efforts to improve the allergy assessment, more information contributing to illness severity, and more accurate pathogen risk assessment. Utilization of ePneumonia increased from 63% during the study period to 90% over the following year with subsequent iterations. Future work includes the adaptation of ePneumonia to hospital EDs in our system, including rural and urgent care settings, exploration of CDS design that provides diagnosis support, and automating the collection of qualitative data from the interaction. Our study demonstrates the feasibility of CDS to engage physicians at the bedside, empower them to share their clinical and CDS experiences, and leverage those experiences to improve CDS design and pneumonia care across the system.
Conclusion
In a study aimed to understand reasons for rejection of best-practice recommendations for pneumonia during the implementation of a computerized CDS, we found that the majority of physicians provided reasons for rejection using unstructured text. Thematic analysis of text data revealed new reasons for rejection not previously highlighted in the literature, including alternative diagnoses to pneumonia and additional patient information used for decision-making not extracted by the tool. CDS implementation that promotes and examines physician–CDS interactions is feasible and provides insights to the implementation and adaptation process across a health care system.
Clinical Relevance Statement
Clinical Relevance Statement
During implementation of a computerized clinical decision support (CDS) tool for pneumonia, we found that the majority of physicians rejecting best-practice recommendations in practice provided reasons for rejection using unstructured text. Thematic analysis of text data revealed alternative diagnoses to pneumonia and additional patient information used for decision-making not specified in best-practice guidelines. CDS that engages physicians in a dialogue is feasible and provides insights to the implementation and adaptation of pneumonia guidelines across a learning health care system.
Multiple Choice Questions
Multiple Choice Questions
Which of the following statements about pneumonia is true?
There is little evidence to guide pneumonia diagnosis and management.
Pneumonia diagnosis and management is very straightforward, with minimal variation.
Pneumonia management demonstrates widespread variation in antibiotic selection and site-of-care decisions.
Clinical decision support tools have dramatically reduced variation and improved practice and outcomes for patients with pneumonia.
Correct Answer: The correct answer is option c. While evidence-based guidelines are associated with improved outcomes, widespread variation in antibiotic selection and site-of-care decisions exist. Clinical decision support tools, are a promising way to reduce the gap between evidence and practice, but they have not been consistently shown to dramatically impact practice.
Which of the following statements about the results from this study is true?
Physicians rejected recommendations from the CDS most of the time.
Physicians who rejected recommendations from the CDS left reasons for adoptions.
Physicians found the CDS difficult to use.
Physicians used the CDS for all pneumonia patients.
Correct Answer: The correct answer is option b. Physicians often agreed with recommendations, but when they rejected recommendations from the CDS, they left reasons for adoptions most of the time.
