Keywords
medication shortage - quality of care - cost of care - electronic health records
Background and Significance
Background and Significance
Across the United States, medication shortages have been increasing over the last few decades.[1] The reasons for these shortages are complex, but can suggest a disruption in the supply chain (e.g., intravenous fluids shortage after Hurricane Maria in 2017),[2] decreased production,[3] change in production priorities by pharmaceutical companies,[4] or increased demand for a particular medication (e.g., increased demand for norepinephrine and hydroxychloroquine during the coronavirus 2019 pandemic).[5]
[6] The types of medication shortage and duration of the shortage vary. During periods when medications are in short supply, patients are more likely to experience adverse events, including mortality and medication errors.[7] In this setting, clinicians must decide how to allocate the limited supply of medication and consider when an alternative medication can be administered instead.
Computerized physician order entry and clinical decision support (CDS) systems are designed to facilitate care delivery by presenting ordering providers with relevant and timely information to enhance quality of care.[8]
[9]
[10] Though these systems have perhaps most often been used to promote adherence to medication treatment guidelines or best practices, they can also be deployed to convey information about short-term, high acuity clinical situations, such as periods of medication shortage.[11]
[12]
[13]
[14] One form of CDS used in such settings are medication shortage alerts (MSAs), which provide specialized guidance to encourage conservation of resources in a responsible way by suggesting alternative medications or treatments. Given the urgency of the medication shortages, MSAs are usually rolled out in a way that would maximize their effectiveness. For instance, these alerts may be designed to be visually distinct from other medication-related CDS alerts, can occur at different points in the ordering process, and their implementation is often accompanied by widespread messaging (e.g., system-wide emails about the ongoing medication shortage, news coverage, training through grand rounds and other educational meetings, etc.; see [Fig. 1]).[15] While medication-related CDS is designed with good intentions, provider response rates are inconsistent with variation among providers[16]
[17]
[18]
[19] and concerns exist that too many alerts within the electronic health record (EHR) may contribute to alert fatigue.[20]
[21]
[22]
Fig. 1 Example medication shortage alert displayed prior to order finalization. (Note: this image is copyrighted by Epic Systems Corporation 2021.)
While there are many studies focused on CDS alerts, data are scarce on the efficacy of alerts that are only active for a short time to address a temporary situation, as well as the susceptibility of these short-term alerts to alert fatigue. In this study, we examined CDS alerts designed specifically for medication shortages (MSAs) to characterize and assess provider behavior in response to these short-term clinical situations.
Materials and Methods
We conducted a retrospective analysis of the usage of inpatient MSAs where an alternative medication was able to be selected and that were active for 60 or more days during the 2-year study period, January 1, 2018 to December 31, 2019, in a large health care system. We chose 60 or more days to allow for enough time for a variety of ordering providers to encounter the alert. Medications administered in the inpatient setting are ordered through the EHR and the medication orders, regardless of ordering provider type, are reviewed and verified by a pharmacist. We examined ordering provider behavior in response to MSAs and characterized their responses. Through a query of the enterprise data warehouse, we identified all MSAs that met this inclusion criterion. Using audit logs and order entry data, we identified each order entry event that resulted in activation of an MSA and recorded the final choice made by that provider. The outcome choices were categorized as: (1) proceeding with ordering the original medication, (2) selecting an alternative medication suggested by the MSA, or (3) abandoning the originally intended medication order altogether ([Fig. 2]). We used these data to characterize provider response to MSAs and examine how provider behavior changed over the study period and across the lifetime of the alert.
Fig. 2 Schematic of workflow with example clinical scenarios.
Ordering providers often encountered individual alerts multiple times. For each alert, we examined the proportion of times each provider accepted the alternative for the alert, abandoned the order, or rejected the suggested alternative. We used descriptive analytics (e.g., graphs and tables) to visualize the trends of providers' response to an alert over time. For each MSA, we characterized the ordering providers who interacted with it at least two times into three distinct groups: (1) accepters – those who accepted an alternative suggested by the MSA more than two-thirds (≥ 67%) of the time, (2) rejectors – those who accepted an alternative less than one-third (≤ 33%) of the time, and (3) flexible – those accepting an alternative suggested by the alert between 33 and 67% of the time.
For each ordering provider, we extracted characteristics about the ordering provider, including sex and provider type. Using the average acceptance rate to an alert for each provider, we then developed a linear regression model to predict average provider response to alerts with the characteristics of the ordering provider (sex, age, provider type) and alerts. In sensitivity analyses, we included additional factors with respect to orders, including order time of day (night shift vs. day shift)[23] and clinical service (surgical vs. nonsurgical). Separately in additional sensitivity analyses, we averaged the provider response categories across all alerts and categorized the frequency of each alert, grouping them into low (the third of alerts with the lowest frequency), medium (the third of alerts with the next lowest frequency), and high (the third of alerts with the highest frequency), and included the categorization in the sensitivity analyses. We also used data from the first time a provider encountered an alert (i.e., whether they accepted the alert during the first exposure) to inform a regression model. Only those providers who were residents, fellows, attendings, nurse practitioners, or physician assistants were included in the regression analysis. In our main analyses, we used a linear regression model recognizing that there may be justification to use a nonlinear approach. Then, for sensitivity analyses, we employed a negative binomial regression and found that differences in provider types were no longer significant. The standard errors were large, however, and thus we used the linear approach for our main analyses. Further, we considered employing an alternative approach that included examining “order sessions” to evaluate if there were differences when orders were placed and signed together. However, from prior work we know that the average number of orders per session is low (median and mean of less than five orders in the emergency department),[23] making it difficult to observe meaningful differences. All analyses were conducted in R 3.6.1. The Mass General Brigham Institutional Review Board approved this study.
Results
During the study period, there were 67 MSAs in use that focused on 42 distinct medications in shortage (see list in [Supplementary Table S1], available in the online version). The MSAs suggested an average of 3.9 alternative medications and were active for an average of 249 days (range 62–728 days). Of the 67 alerts studied, there were 4 in which no provider continued with the original order. The frequency with which alerts were triggered diminished after 30 days but remained stable for the remaining time of the alert. Twenty percent of activations of the alerts were in the first 30 days after an alert became active for the first time. This decreased to 18% over the next 30 days, as well as ensuing 30-day periods.
A total of 214,373 orders were placed that prompted an MSA. There were 8,917 unique providers who encountered at least one of the MSAs and 5,553 who encountered at least one MSA more than once ([Table 1]). During the study period, ordering providers responded to the alert by choosing a suggested alternative medication 17.2% of the time.
Table 1
Characteristics of ordering providers who encountered one MSA at least twice
|
Attending physician
|
Fellow
|
Resident
|
Nurse practitioner
|
Physician assistant
|
|
Total unique ordering providers
|
3,082
|
298
|
996
|
652
|
525
|
|
Female (%)
|
45.7%
|
45.6%
|
47.5%
|
94%
|
79.2%
|
|
Total orders placed
|
84,784
|
14,205
|
55,519
|
15,322
|
28,105
|
|
Average number of MSAs encountered
|
3.4
|
6.9
|
8.3
|
3.2
|
6.4
|
|
Proportion who consistently accepted MSAs (%)
|
5.5%
|
2.3%
|
3.8%
|
1.7%
|
4.2%
|
|
Proportion who consistently rejected MSAs (%)
|
62.1%
|
43.6%
|
33.1%
|
61.7%
|
42.9%
|
|
Proportion who varied in response to MSAs (%)
|
32.4%
|
54%
|
63.1%
|
36.7%
|
53%
|
Abbreviation: MSA, medication shortage alert.
For those providers encountering an MSA at least twice, most ordering provider responses were to reject the suggestion and disregard the alternative medication (3,000, 54.0%), although some providers did show varied responses across different alerts. A small fraction of providers tended to consistently accept the alternatives suggested by nearly all MSAs (251, 4.5%). The remaining 41.5% changed their response across alerts but tended toward more frequent rejection.
Adjusting for the different alerts, fellows (estimate = –0.016, 95% confidence interval [CI] = –0.026, –0.005, p = 0.004), residents (estimate = –0.007, 95% CI = –0.014, –0.001, p = 0.03), and physician assistants (estimate = –0.010, 95% CI = –0.019, –0.002, p = 0.02) were less likely to accept alerts on average compared with attending physicians. Female ordering clinicians were more likely than male ordering clinicians to accept alerts (estimate = 0.012, 95% CI = 0.006–0.018, p < 0.001) ([Table 2]). Results were similar when incorporating order time of day and clinical service as additional covariates into the model. Results were similar when incorporating the alert frequency groupings rather than the alerts and demonstrated that the highest frequency alerts were least likely to be accepted (estimate = –0.27, 95% CI = –0.31, –0.23, p < 0.001) compared with low or medium frequency alerts.
Table 2
Linear regression results of ordering provider response to alerts
|
Variable
|
Estimate
|
95% confidence interval
|
p-Value
|
|
Female ordering clinician
|
0.012
|
|
< 0.001
|
|
Provider type
|
|
Attending physician
|
reference
|
reference
|
reference
|
|
Fellows
|
–0.016
|
(–0.026, –0.005)
|
0.004
|
|
Residents
|
–0.007
|
(–0.014, –0.001)
|
0.03
|
|
Physician assistants
|
–0.010
|
(–0.019, –0.002)
|
0.02
|
Note: Linear regression controlled for different alert types. Please note, this analysis was limited due to the inability to control for other confounding factors (e.g., ordering clinician clinical service). Low efficacy of medication shortage clinical decision support alerts.
We then examined the impact of an ordering provider's first response to a specific alert on their overall acceptance of alerts, adjusting for provider type, sex, and alert. We found that those who were more likely to accept an alert the first time they encountered it were significantly more likely to accept more alerts on average (estimate = 0.422, 95% CI = 0.415–0.429, p < 0.001).
Discussion
In our study, CDS alerts designed to address short-term situations were accepted by providers only about one time in six. Across the lifetime of the alert, the acceptance rate was less than 20%. For some medications, for example, if the inventory level of the medication is at a critical low or is truly not available, “hard stop” alerts can be effective.[24]
Our findings demonstrated that providers most often rejected MSAs, even those who were sometimes flexible about their responses. The low overall acceptance rate supports the theory that alerts for MSAs appearing at the time of order entry may have limited value, as they may be presented too late in the clinical decision-making process, particularly if they alert after the decision has been made.[25]
[26]
MSAs are designed with the assumption of high efficacy, particularly as they are implemented during a period of stress for the health care system. They may be visually distinct from other CDS alerts and their implementation is often accompanied by widespread messaging (e.g., system-wide emails about the ongoing medication shortage, news coverage). Because MSAs are short term, we would not expect those providers who continued to order alert-triggering medications to develop “muscle memory” to ignore them.
We found that fellows, residents, and physician assistants were less likely to accept the alternative when compared with attending physicians, though the coefficients were small. This suggests that attending physicians may be more open to changing their practice than some other provider types, who may have fixed ideas about how to treat their patients, not have the authority to make the change, or be less confident in prescribing a suggested medication that might be new. Given that we were unable to control for patient location or clinical service, these findings may also indicate patient-level variation, particularly if attending physicians work with different types of patients than trainees or physician assistants. Regardless, the clinical significance of these differences should be further evaluated.
Our study has several limitations. First, we were unable to evaluate characteristics of the patients or clinical situations when the alert was activated to determine the optimal response to the alert. Second, this study did not have a control group to determine how much the alert itself changed overall behavior. Third, we queried the institutional Enterprise Data Warehouse for these specialized CDS alerts which may not fully capture all MSAs in our institution's EHR vendor, Epic Systems. Fourth, we were not able capture the individual site-specific medication supply or the site-specific decisions for utilizing the MSAs during the time of the medication shortages. Fifth, we were not able to determine the extent to which communication outside of the EHR and CDS alerts (e.g., emails about medication shortage) impacted provider workflow. Sixth, we were unable to account for all potential confounding variables, limiting the conclusions associated with alert rejection. Lastly, the approach we used for combining the acceptance rates is rudimentary as it only looks at point estimates for each provider. A more sophisticated approach incorporating weighted averages or shrinkage methods (such as the EB05 method used in pharmacovigilance)[27] could be used in future research to validate the robustness of these findings.
Conclusion
Though MSAs are designed to be attention-grabbing and higher impact than traditional CDS, our findings suggest that providers rarely change their clinical decisions when presented with these alerts. However, it appears that some providers did learn to avoid certain medications that would trigger these types of alerts, decreasing the frequency of the alerts after the first 30 days. The overall low acceptance rate, however, suggests the need to consider innovative ways to improve these alerts and the response rates to them.
Clinical Relevance Statement
Clinical Relevance Statement
Medication shortage alerts are designed with the assumption of high efficacy, particularly as they are implemented during a period of stress for the health care system. Our findings suggest that providers rarely change their clinical decisions when presented with these alerts. The overall low acceptance rate suggests the need to consider innovative ways to improve these alerts and the response rates to them.
Multiple Choice Questions
Multiple Choice Questions
-
What are medication shortage alerts?
-
Alerts sent to patients regarding their current prescriptions and need to refill.
-
Alerts that provide specialized guidance to encourage conservation of resources in a responsible way by suggesting alternative medications or treatments.
-
Alerts that the ordering provider is prescribing a medication at too low a dose or quantity for the patient.
-
Alerts that the patient's local pharmacy is out of a medication.
Correct Answer: Option b is the correct answer. Computerized physician order entry (CPOE) and clinical decision support (CDS) systems are designed to facilitate care delivery by presenting ordering providers with relevant and timely information to enhance quality of care. Though these systems have perhaps most often been used to promote adherence to medication treatment guidelines or best practices, they can also be deployed to convey information about short-term, high acuity clinical situations, such as periods of medication shortage. One form of CDS used in such settings are medication shortage alerts (MSAs), which provide specialized guidance that encourage conservation of resources in a responsible way by suggesting alternative medications or treatments.
-
After adjusting for sex of the ordering provider and the different alerts, when compared with attending physicians, what types of clinicians were less likely to accept medication shortage alerts?
-
Fellows
-
Residents
-
Physician assistants
-
All the above
Correct Answer: Option d is the correct answer. Adjusting for sex of the ordering provider and the different alerts, fellows (estimate = –0.016, 95% confidence interval [CI] = –0.026, –0.005, p = 0.004), residents (estimate = –0.007, 95% CI = –0.014, –0.001, p = 0.03), and physician assistants (estimate = –0.010, 95% CI = –0.019, –0.002, p = 0.02) were less likely to accept alerts on average compared with attending physicians.
