Impact of a Vendor-Developed Opioid Clinical Decision Support Intervention on Adherence to Prescribing Guidelines, Opioid Prescribing, and Rates of Opioid-Related Encounters

Keywords

clinical decision support system - electronic health record - opioids - quality improvement

Background and Significance

Deaths from opioid overdose in the United States have risen dramatically in the past decade.[1] Missouri has been among the states harder hit, with a death rate of 16.5 per 100,000 persons, above the national average of 14.6 deaths per 100,000.[2] While deaths from prescription opioid use decreased slightly from 2017 to 2018, opioid prescription misuse continues to account for a significant portion of opioid-related mortality.[3]

Provider prescribing practices related to pain are felt to be an important factor in the rise of opioid-related deaths.[4] Centers for Disease Control and Prevention (CDC) and other groups have issued guidelines to support evidence-based practice and improve prescribing practices and patient safety.[5] [6] [7] However, clinician adoption of evidence-based practices and adherence to treatment guidelines is suboptimal due in part to the large number of recommendations and the lack of clinical trials and other evidence to support their effectiveness in reducing opioid misuse and related outcomes.[8] [9] Clinical decision support systems (CDSS) can further support evidence-based opioid-prescribing practices, and studies of opioid CDSS may enhance the evidence basis for guidelines. As a component of a comprehensive approach to improving opioid prescribing, CDSS can contribute to significant improvements in guideline adherence and opioid-prescribing practices.[10] [11] [12] The role and importance of opioid CDSS alone is less clear.

Objectives

To improve prescribing practices, promote patient safety, and assure adherence to clinical practice guidelines, the University of Missouri implemented a package of vendor-developed, provider-targeted opioid CDSS interventions. The aim of this study was to describe the impact of the implementation on guideline adherence, opioid prescribing, and opioid-related encounters.

Methods

The University of Missouri is an academic health system with five hospitals and over 60 clinic locations across central Missouri. As part of a quality improvement project, the University of Missouri implemented an opioid CDSS originally developed by our electronic health record (EHR) vendor (Cerner Corp., Kansas City, Missouri, United States). The “out of the box” components included five alerts and one form in which users enter metadata related to the opioid treatment acknowledgment (OTA) such as start date, primary and secondary prescribers, and preferred pharmacy. The package also included rules that codified the business logic behind the alerts. The alerts fired based on the metadata in the form and opioid order entry actions by the provider. Institutional experts in pain management and clinical informatics reviewed the components prior to implementation and made localizations to improve conformance with common usability heuristics[13] and reduce overalerting. The following localizations were made: (1) a status field was removed from the form to prevent users from entering a status discordant with OTA dates on the form, (2) a cancel button was added to the form, (3) links to the OTA and form were added to some alerts, (4) some alerts were re-titled, (5) alerts were suppressed for patients with an active cancer diagnosis defined by any child code in the SNOMED CT hierarchy under 363346000 on the problem list, and (6) alerts were suppressed if the prescription came from a palliative care location or specialist. Override functionality was implemented as delivered by the vendor. Five provider-facing, interruptive order entry alerts were implemented. Three alerts encouraged the establishment of and adherence to OTA ([Fig. 1]). Two other alerts focused on high-risk patients and prescriptions, recommending the use of naloxone and urging caution with high-risk patients in a manner congruent with CDC guidelines for prescription opioid use ([Fig. 2]).[5] Conditions triggering these two alerts include prescription of extended release opioids in opioid-naïve patients, morphine milligram equivalent (MME) doses over 50, concurrent benzodiazepine use, medical problems that increase patient overdose risk, and past positive drug screens. The vendor alerts fired for patients at least 18 years of age, in all locations except for the cancer center, and for all emergency department and outpatient encounters. These vendor alerts did not fire for opioids administered in the emergency department but did fire for discharge prescriptions. Alerts did not fire for prescriptions written on inpatient, observation, or ambulatory surgery encounters or for patients receiving palliative or hospice care. In response to a new state statute,[14] an additional alert recommending a duration less than 7 days for initial opioid prescriptions for acute pain was developed and implemented at week 44. This alert fired in all settings of care but did not fire for patients receiving palliative care or hospice services. Providers received educational documents describing the alerts and workflows prior to the implementation. Altogether, one form and six alerts were implemented system wide ([Table 1]).

Fig. 1 Opioid Treatment Acknowledgment—unlisted prescriber alert. This alert fired when ordering an opioid if the prescriber was not listed as primary or secondary prescriber on the form.

Fig. 2 Opioid high-risk alert. This alert was retitled for clarity. Note that alert actions do not include an override, and override data were not available from the vendor.

Table 1 Components of the opioid clinical decision support system implemented at the University of Missouri

Name	Description	Override available	Available alert actions	Localization
OTA form	Used for documentation of OTA start date, OTA expiration date, primary prescriber, secondary prescriber, and preferred pharmacy	N/A	N/A	Status field was removed from the opioid treatment agreement form; cancel button added
OTA—unlisted prescriber	Alerts on opioid order entry if the prescriber is not recorded as the primary or secondary prescriber	Yes	Remove order or override	None
OTA—missing agreement	Alerts on opioid order entry if problem “opioid dependence with current use” present on the problem list and an OTA has not been recorded or has expired	Yes	Remove order or override	Retitled; added links to the OTA and OTA form.
Long-term opioid therapy	Alerts on opioid order entry if the patient has had opioids on the majority of days in the past 3 mo and OTA has not been recorded or has expired	No	None	Retitled; added link to the OTA.
Opioid high-risk	Alerts on opioid order entry when conditions exist which increase patient's risk of adverse events related to opioids (MME > 50; concurrent benzodiazepine; extended release opioids in opioid naïve; three or more opioid prescriptions in the past 30 d; various comorbid conditions)	No	Cancel prescription or continue	Retitled
Naloxone	Alerts on opioid order entry when prescribing opioids, naloxone has not been prescribed, and conditions exist which increase patient's risk of adverse event related to opioids (MME > 50; concurrent benzodiazepine; extended release opioids in opioid naïve; three or more opioid prescriptions in the past 30 days; various comorbid conditions)	No	None	None
Initial prescription for acute pain	Alerts on opioid order entry if the initial opioid prescription duration exceeds 7 days	No	Hard stop, requires a reason for an extended prescription >7 days	N/A

Abbreviations: MME, morphine milligram equivalent; N/A, not available; OTA: opioid treatment acknowledgment.

The Lights On Network is a collection of cloud-based data analytics tools provided by Cerner Corporation that monitor clinical activity, user experience, and other system metrics and served as one source of data for the study. Cerner developed opioid performance metrics concurrently with the opioid CDSS package and makes these metrics available to all clients regardless of their use of the opioid CDSS. We included these metrics as outcomes to determine how well vendor-supplied data reflected actual opioid prescribing and encounter outcomes derived directly from our health system database. From Lights On Network were obtained, by week, prescribing outcome of the percent of opioid-naïve patients receiving short-acting opioids starting week 6, and the guideline adherence outcomes of percent of high-risk patients prescribed opioids who had naloxone provisioned starting week 11 and percent of chronic opioid users with a current opioid treatment agreement starting week 38. Lights On Network outcomes were not constrained by age. The intervention itself was a standardized way of capturing and documenting opioid treatment agreements; therefore, no pre-implementation data were available for this guideline adherence measure.

Prescribing and encounter data 30 weeks before and after implementation of the opioid CDSS (between November 21, 2018 and January 16, 2020) were extracted from the University of Missouri Cerner Millennium client database. We chose outcomes based on what we envisioned from a successful opioid CDSS implementation: fewer opioid prescriptions, smaller doses and quantities, fewer high-risk combinations such as opioids with benzodiazepines, all with a goal of fewer opioid-related encounters. From the Millennium database were derived, by week (1) total opioid prescriptions; (2) average MME per prescription; (3) percent of opioids prescribed concurrently with a benzodiazepine; (4) encounters for opioid abuse and dependence; and (5) encounters for opioid overdose and poisoning. We normalized total opioid prescriptions and opioid-related encounters by total health system encounters, expressed as prescriptions or encounters per 1,000 total health system encounters to minimize confounding by total health system patient volume. MME per prescription was used instead of MME per day because daily MME cannot be accurately calculated for as needed (PRN) medications. MME was calculated by multiplying the dispensed amount by established MME conversion factors.[15] An encounter was classified as abuse and dependence or overdose and poisoning if the encounter was coded with an International Classification of Diseases, Tenth Revision, Clinical Modification code found in the value set from the Council of State and Territorial Epidemiologists of the same names found at the Value Set Authority Center, U.S. National Library of Medicine.[16] To align with outcomes from the Lights On Network, prescription and encounter outcomes derived from the Millennium database were not constrained by age. Encounter counts were not limited to patients receiving opioid prescriptions. Buprenorphine prescriptions were excluded from the opioid-prescribing outcomes because this medication is used to treat opioid use disorder. Prescriptions and encounters were excluded if the patient was on a palliative care service or if the prescriber was in a palliative care specialty. Finally, we calculated weekly firing rates for each alert.

We calculated descriptive statistics, including prescription recipient mean age and self-reported gender. We also describe prescribers by their position (attending, fellow or resident, and advanced practice provider), and whether they were primary care providers (PCP). We calculated prescribers per patient and pharmacies used per patient before and after the opioid CDSS implementation. Student's t-test and chi-square tests were used for continuous and categorical descriptive variables, respectively.

For the main analysis, we used an interrupted time-series (ITS) analysis over the 30 weeks period before and after implementation of the opioid CDSS, from November 21, 2018 to January 16, 2020. Linear regression models were fitted to the time-series data using the form:

Y_t  = β ₀ + β ₁ T_t  + β ₂ X_t  + β ₃ X_tT_t  + ε_t

In this model, Y_t is the outcome variable, T_t is the time variable since the start of the study, and Xt is a dummy variable equal to 0 for the preintervention period and 1 for the postintervention period. β₀ is the baseline level at the beginning of the study period, β₁ is the slope of the preintervention portion of the regression line, β₂ is the immediate change in the regression line with the intervention, and β₃ is the change in the slope of the regression line postintervention.[17] Both the immediate intervention effect (β₂ ) and the effect of the intervention over time as measured by the change in the slope of the regression line postintervention (β₃ ) were evaluated to determine the impact of the CDSS intervention. We estimated coefficients using ordinary least squares regression with Newey–West standard errors adjusted for autocorrelation and no adjustments for seasonality or nonstationarity.

We considered various ITS controls.[18] No location-based, characteristic-based, or behavior-based control groups were available. Control outcomes such as prescriptions for nonsteroidal anti-inflammatory medications, gabapentinoids, and antidepressants were unlikely to be prescribed to the same groups of patients as opioid medications, and none of the control outcomes considered were subject to the same regulatory scrutiny and prescribing pressures as were opioids. Therefore, we used no control outcomes and instead performed a single treatment period, single-group ITS analysis.

Data from the Cerner Millennium database afforded the opportunity to perform subgroup analyses on three of the prescribing outcomes. Because the alerts from the vendor only fired on those patients of 18 years of age and older, we performed single-group ITS analysis of opioid prescriptions/1,000 encounters, MME per prescription, and percent opioids with concurrent benzodiazepine use on the subset of prescriptions for patients at least 18 years old at the start of the study. To assess the impact of the intervention on those patients getting more prescription fills and fills for higher MME doses, we performed the same analysis on the subset of prescriptions in the top quartile of MME and for those patients who received more than one opioid prescription during the study period. Lastly, we performed multigroup ITS comparison by prescriber position (trainee vs. attending, advance practice provider [APP] vs. attending) and by prescriber role (PCP vs. others). Statistical significance was defined at α = 0.05. The statistical analysis was performed using Stata/IC v16.1 (College Station, Texas, United States) with packages st0389_7 and actest. We followed the applicable portions of the SQUIRE 2.0 framework for reporting quality improvement activities.[19] The University of Missouri institutional review board determined that the project was quality improvement activity, not human subject research, and did not require additional IRB review.

Results

Alert Firing

Implementation of the opioid CDSS resulted in the addition of on average 1,656 (interquartile range 1,605–1,814) alerts per week. The alerts cautioning the prescriber about high-risk patients and those recommending naloxone comprised the majority of alerts fired ([Fig. 3]). The opioid CDSS resulted in an average of 6.3 (interquartile range 6.1–7.2) additional alerts per opioid prescriber per week post-implementation. Override rates for the two overridable alerts were 76.3 and 89.4% for the alert declaring that the prescriber was not listed on the OTA and the alert declaring that an OTA was missing or expired, respectively.

Fig. 3 Firing rates for the six opioid clinical decision support alerts. OTA: opioid treatment acknowledgment.

Prescribing Outcomes

Total opioid prescriptions numbered 34,678 among 14,355 unique patients during the pre-implementation period and 31,732 among 13,030 unique patients during the post-implementation period. Of the unique patients with an opioid prescription prior to implementation, 4,456 (31.0%) received an opioid prescription post-implementation. After implementation, patients were slightly older (53.0 vs. 52.7 years, p = 0.021) and prescribers were somewhat more likely to be PCP (51.5 vs. 49.8, p < 0.001, [Table 2]). The number of prescribers from whom patients received prescriptions, 2.4, did not differ pre- and post-implementation (95% confidence interval [CI] for difference:−0.5, 0.9; p = 0.58). Patients used on average 1.33 pharmacies preintervention and 1.50 pharmacies postintervention (95% CI for difference: 0.10, 0.24; p < 0.001).

Table 2 Characteristics of prescriptions by patient and provider

	Preintervention	Postintervention	p-Value
Prescriptions	34,678	31,732
Unique patients	14,355	13,030
Prescriptions by patient characteristics
Mean age (years)	52.7	53.0	0.021
Female	19,824 (57.2%)	18,434 (58.1%)	0.053
Prescriptions by prescriber characteristics
Attending	22,148 (63.9%)	20,193 (63.6%)	0.425
Resident or fellow	7,253 (20.9%)	6,764 (21.3%)
Advanced practice provider	5,277 (15.2%)	4,775 (15.1%)
Primary care	17,255 (49.8)	16,337 (51.5%)	<0.001

The slope representing opioid prescriptions over time prior to the opioid CDSS implementation was negative, representing a decline in the number of opioid prescriptions per 1,000 encounters per week (−0.175 [95% CI −0.278, −0.072]). This weekly decrease in opioid prescriptions slowed significantly post-implementation. The drop in opioid prescriptions continued but at a slower rate, signified by a significant change for the worse in the post-implementation slope (+0.161 [95% CI 0.014, 0.308]). Otherwise, we found no immediate change (β₂ ) or change over time (β₃ ) between the pre- and post-implementation periods in any prescribing outcome ([Table 3], [Fig. 4]). The vendor-supplied prescribing outcome, percent of opioid-naïve patients prescribed a short-acting opioid, was consistent with the other prescribing outcomes derived from directly from the EHR. The number of opioid prescriptions with concurrent benzodiazepines decreased over the study period with no significant alteration of that trend with the opioid CDSS implementation at week 31. Lights On Network found an average of 737 opioid prescriptions for opioid-naïve patients per week. The proportion of these patients receiving a short-acting opioid was high across the study period, generally over 95%, and did not change with the opioid CDSS intervention.

Table 3 Single-group ITS model coefficients [95% confidence interval] for prescribing, encounter, and guideline adherence outcomes

	Baseline (β0 )	Preintervention slope (β1 )	Immediate change (β2 )	Postintervention slope	Slope change (β3 )
Prescribing outcomes
Opioid prescriptions/1,000 encounters[a] [f]	32.3 [30.3, 34.3]	−0.175 [−0.278, −0.072][g]	0.52 [−1.49, 2.53]	−0.014 [−0.119, 0.091]	0.161 [0.014, 0.308][g]
MME/opioid prescription[a]	813 [774, 852]	−1.222 [−3.571, 1.128]	15.9 [−41.7, 73.6]	0.079 [−2.098, 2.257]	1.301 [−1.902, 4.504]
Concurrent benzodiazepine/opioid prescription (%)[a]	2.78 [2.50, 3.06]	−0.017 [−0.035, 0.000]	0.37 [−0.10, 0.85]	−0.019 [−0.039, 0.000]	0.000 [−0.028, 0.024]
Short-acting opioids in opioid naïve (%)[b] [c]	96.4 [95.7, 97.0]	0.002 [−0.047, 0.042]	−0.04 [−0.83, 0.75]	0.011 [−0.15, 0.038]	0.014 [−0.038, 0.066]
Encounter outcomes
Abuse and dependence/1,000 encounters[a] [f]	0.446 [0.352, 0.540]	0.000 [−0.006, 0.006]	−0.065 [−0.182, 0.051]	0.002 [−0.001, 0.006]	0.002 [0.005, 0.009]
Overdose and poisoning/1,000 encounters[a]	0.081 [0.041, 0.120]	0.001 [−0.002, 0.003]	−0.004 [−0.054, 0.047]	0.001 [−0.001, 0.004]	0.001 [−0.003, 0.004]
Guideline adherence outcomes
Provision of naloxone (%)[b] [d] [f]	0.674 [0.441, 0.908]	0.049 [0.019, 0.078][g]	1.13 [0.32, 1.93][g]	0.069 [0.036, 0.101][h]	0.020 [−0.022, 0.063]
Opioid treatment acknowledgment (%)[b] [e]	N/A	N/A	N/A	0.369 [0.261, 0.477][h]	N/A

Abbreviations: ITS, interrupted time series; MME, morphine milligram equivalents; N/A, not available.

^a Millennium database.

^b Lights On Network.

^c Starting week 6.

^d Starting week 11.

^e Starting week 38.

^f First-order (lag 1) autocorrelation.

^g p < 0.05.

^h p < 0.001.

Fig. 4 Scatterplots showing opioid prescribing, encounter, and guideline adherence CDSS outcomes, by week. Vertical line represents the beginning of the intervention.

Encounter Outcomes

Opioid abuse and dependence encounters averaged 17.2 and 16.2 per week in the pre- and post-implementation periods, respectively. After adjustment for total encounter volume, rates of encounters for opioid abuse and dependence did not change with opioid CDSS implementation. Opioid poisoning and overdose encounters averaged 3.5 and 4.2 per week pre- and post-implementation, respectively, and similarly were unchanged after opioid CDSS implementation.

Guideline Adherence Outcomes

Lights On Network identified an average of 784 prescriptions per week to patients using chronic opioids who were candidates for opioid treatment agreements. The proportion of prescriptions in the setting of a documented opioid treatment agreement increased throughout the 30 weeks post-implementation period, from under 5 to over 15% by the end of the study period (change in slope +0.369% per week [95% CI 0.261, 0.477], [Table 3], [Fig. 4]). An average of 1,074 prescriptions per week were tied to either a dose or medical conditions that increased the risk of overdose. These patients were considered candidates for naloxone. An increase in the rate of provision of naloxone was noted throughout the study period, but a significant immediate improvement was found with the intervention (+1.13% [95% CI 0.32, 1.93]; p = 0.007), then gradually improving to nearly 5% of high-risk patients.

Subgroup Analyses

Analysis of opioid prescription counts, MME per prescription, and concurrent benzodiazepine use among the subset of patients of 18 years of age and older revealed findings similar to the primary analysis; the rate of decrease in the opioid prescription count slowed, but we found no other immediate change (β₂ ) or change over time (β₃ ) between the pre- and post-implementation periods ([Supplementary Table S1], available in the online version). Among the subset of patients with more than one opioid prescription during the study period, we found no changes in opioid prescription counts, MME per prescription, and concurrent benzodiazepine use with the opioid CDSS intervention. Similarly, among the subset of prescriptions in the top quartile of MME, these same outcomes did not change ([Supplementary Table S2], available in the online version). Subgroup analysis by prescriber role demonstrated an immediate increase in the rates of concurrent benzodiazepine prescribing with the CDSS among non-PCP. MME per prescription was significantly higher for PCP, but we found no other significant differences or changes with opioid CDSS implementation between PCPs and others ([Supplementary Table S3], available in the online version). Subgroup analysis by provider position showed that at baseline, trainees wrote significantly fewer prescriptions and lower MME per prescription than did attendings. Compared with attendings, at baseline, APPs wrote fewer opioid prescriptions, lower MME per prescription, and had lower concurrent benzodiazepine use than attendings. The attending subgroup demonstrated no significant immediate change (β₂ ) or change over time (β₃ ) with the intervention, and the APP and trainee subgroups did not differ significantly from the attendings as a result of the intervention ([Supplementary Table S4], available in the online version).

Discussion

The opioid CDSS resulted in improvement in some measures of adherence to relevant opioid-prescribing guidelines. CDC guidelines recommend consideration of naloxone with an opioid prescription when high-risk conditions are present.[5] Naloxone access laws improve naloxone availability[20] but such laws vary from state to state. In 2016, Missouri pharmacists were authorized to dispense naloxone without a prescription under protocol with an authorizing physician. No additional regulations were enacted in Missouri during the study period. We noted a trend toward increasing rates of naloxone provisioning prior to the intervention, consistent with trends for the provision of naloxone across the country.[21] In our study, the rate of naloxone provision increased throughout the study period but jumped significantly with the CDSS intervention. At the beginning of the study period, naloxone accompanied opioid prescriptions for fewer than 1% of high-risk patients, but by the end of the study that figure had risen to almost 5%. Since the end of the study, naloxone continues to be provided at a rate of over 6%. This rate is higher than the national average in 2018 that stood at 1.5%.

Opioid treatment agreements are included in recommendations from the CDC[7] and are frequently found in other guidelines,[6] although the evidence for OTAs is relatively weak[22] and their use is not without controversy.[23] Treatment agreements were used prior to the intervention but were inconsistently documented and were not stored in the EHR as structured data. We observed an increase in documentation of opioid treatment agreements during the post-implementation period, to over 15% of candidate chronic opioid therapy patients. It is likely that much of this improvement is due to the standardization of documentation built into the CDS intervention and improved data capture consistent with the vendor quality measurement.

We were not surprised to find no significant improvement in opioid-related encounter outcomes. Changes in opioid encounter rates may not manifest within the 30 weeks post-implementation period of this study. Further, CI on the immediate change and change over time in opioid encounter rates were relatively wide. When interpreting a study with negative results such as ours, it is appropriate to consider the minimal clinically important difference (MCID) in intervention outcomes—in this case the minimal clinically important change in rates of opioid encounters—and whether this difference is within or outside of the reported CI.[24] [25] When the MCID is within the reported CI, the data could still be consistent with meaningful benefit or harm. For example, the lower limit of the CI for immediate change in opioid abuse and dependence encounters is −1.82 per 1,000 encounters. Given our health system volume, this equates to an immediate decrease of seven abuse and dependence encounters weekly. One might reasonably conclude that the MCID for opioid abuse and dependence encounters weekly is lower than seven, within the CI and, therefore, that our study has not excluded meaningful benefit from the implementation.

Improvement in encounter outcomes is unlikely without an improvement in prescribing outcomes. The opioid CDSS did not statistically significantly improve any of the prescribing outcomes in our study, and in fact, we observed a statistically significant worsening post-implementation in rates over time of opioid prescriptions per 1,000 encounters. The CI were relatively narrow, making it less likely that an MCID would be within the CI and more likely that a finding of meaningful benefit or harm from the opioid CDSS can be excluded.

When considering the failure to improve opioid-prescribing outcomes, we must consider how well this system conformed to the five “rights” of clinical decision support.[26] The alerts appear on order entry, which is appropriate when a provider placed the order. However, some alerts may not have been presented to the right person, as the alerts also appeared for nurses and medical assistants who can propose medications for the provider to sign later. Some of the alerts are driven by the metadata in the form; entering that data may be viewed as an administrative function, better handled by staff at a different point in the workflow.

Usability issues may also reduce the effectiveness of EHR systems.[27] [28] We made some minor modifications to correct usability concerns we found in the “out of the box” version of the system. These changes were relatively minor in scope. Opioid-prescribing rates might be improved with more significant usability improvements: combining the multiple alerts into a single alert to reduce clicks and offering opioid alternatives directly from the alert to improve flexibility and efficiency of use. Unfortunately, major usability improvements and the associated usability testing are beyond the capacity of most sites implementing the system.[29]

Our study is one of the first to evaluate a vendor-developed opioid CDSS. Systems developed by vendors have advantages over “home-grown” solutions. They are relatively easy to disseminate and implement, and they eliminate the need for health systems to complete the work on their own. Vendors may be quick to promote the value of such systems and may even incentivize health systems to adopt them.[30] [31] However, published studies of vendor-developed CDSS are relatively uncommon, and such studies may not replicate the success claimed by the vendor developer.[32] Careful study of such systems, including replication studies,[33] is needed to assure that vendor-developed CDSS achieves the outcomes advertised. Comparisons of vendor-supplied metrics with health-system-defined measures derived directly from the system database will inform the utility, validity, and relevance of vendor-supplied quality indicators. Greater collaboration between vendors and health systems during feature development will help minimize usability problems and improve the performance of vendor-developed CDSS.

Usability improvements and better conformance to the five “rights” framework may still not be enough to improve opioid-related prescribing and encounter outcomes. Our work adds to the body of evidence suggesting that opioid CDSS alone, while it may improve some measures of guideline adherence, does not appear to improve prescribing practices or opioid-related encounters. A retrospective analysis in Minnesota found no change in the co-prescribing of opioids and benzodiazepines with the implementation of a prescribing alert.[34] A prospective cohort study of an opioid CDS intervention in Louisiana found improved adherence to risk mitigation strategies such as urine drug screening and provision of naloxone but no change in daily MME or hospitalization rates.[35] A study of a multicomponent opioid intervention, including EHR templates and dashboards, improved the use of OTA and urine drug screens but did not alter prescribing.[36] Some studies of multi-faceted interventions, which include not only CDSS but also other elements such as education, auditing and feedback, and formulary limitations, have shown improvements in prescribing practices.[10] [12] The type of CDSS used varies widely across studies and includes alerts, reference links, risk prediction tools, templates, and dashboards. Additional research is needed to determine which combinations of interventions are effective in improving not only guideline adherence but opioid prescribing and opioid-related encounters.

The intervention resulted in a large number of alerts. Some alerts decreased in frequency over the study period, likely in response to the improved capture of OTA that was part of the CDSS. The increased alert burden is associated with provider burnout and risks to patient safety.[37] [38] [39] [40] Opioid CDSS, in particular, have been associated with a high volume of inconsequential alerts.[41] Any increase in alert burden, with the attendant risk of alert fatigue, must be accompanied by measurable improvement in outcomes.

Quasi-experimental ITS models are generally free from confounding due to between-group differences based on the use of aggregated data from a single population.[18] However, results may still be confounded by concurrent interventions such as the added regulatory requirement to limit opioids for acute pain to 7 days supply. This requirement went into effect during the intervention period but was expected to have minimal effect given that (1) it would primarily impact opioid-naïve patients and (2) if anything it would augment the vendor-developed opioid CDSS interventions. Failure to find a significant impact on encounters or prescribing suggests that confounding by this co-intervention is unlikely. Implementation of a state-wide prescription drug monitoring program (PDMP) could confound results as the goal of such programs is to assist prescribers in the identification and prevention of prescription drug abuse. However, the state of Missouri does not have a state-wide PDMP and no PDMP-checking requirement was in place at the time of our study. The study terminated just before the severe acute respiratory syndrome coronavirus 2 pandemic caused widespread health system changes in our state. There were no other major changes in the health system patient or prescriber populations, and we are aware of no other external factors such as payer policies that changed during the study period. Therefore, confounding by other co-interventions is unlikely.

Our study has other important limitations. First, our study evaluated only one component of the opioid package developed by the vendor. Other components not evaluated in this study and still yet to be released and implemented may amplify the beneficial effect of the CDSS on guideline adherence and improve other outcomes. Second, this study was not constrained to patients on chronic opioid therapy. It is possible that a benefit to the opioid CDSS may be seen in these or other subsets of patients. Elucidating those subsets of patients who stand to benefit the most from opioid CDSS will allow more targeted opioid alerting. Third, we did not capture opioid prescribing and encounter outcomes at facilities unaffiliated with our institution. Finally, a single-site study such as ours has limited generalizability, and other environments and localizations may have different results.

Conclusion

A vendor-developed, provider-targeted opioid CDSS improved some measures of provider adherence to opioid-prescribing guidelines but did not improve opioid-prescribing practices or rates of opioid-related encounters. Further work is needed to determine the optimal configuration of opioid CDSS and its role within a multi-faceted program of interventions to reduce opioid prescribing and opioid-related adverse outcomes.

Clinical Relevance Statement

The vendor-developed opioid CDSS studied described here results in a significant number of increased alerts. The CDSS improved adherence to opioid-prescribing guidelines but did not alter opioid prescribing or rates opioid-related encounters.

Multiple Choice Questions

1. Based on this study of opioid CDSS, which of the following is expected to improve with the implementation of an opioid clinical decision support system?

   • Counts of opioid prescriptions

   • Use of short-acting opioids in opioid-naïve patients

   • Rates of provision of naloxone

   • Rates of opioid-related encounters

   Correct Answer: The correct answer is option c, rates of provision of naloxone. In this study, the provision of naloxone improved significantly. The documentation of opioid treatment agreements also improved, but this improvement was likely due to improved data capture built into the decision support system. Counts of opioid prescriptions, the use of short-acting opioids in opioid-naïve patients, and opioid-related encounters did not change.

2. Which of the following types of interventions is likely to be effective in reducing opioid prescribing?

   • Order entry opioid alerts

   • Provider education

   • Provider dashboards

   • Multi-faceted interventions

   Correct Answer: The correct answer is option d, multifaceted interventions. This study of opioid CDSS using order entry alerts did not alter prescribing behavior. Studies that show changes in prescribing practices more often include multi-faceted interventions with elements that might include alerts, education, auditing and feedback, and formulary limitations.

• References

• 1 National Institute on Drug Abuse.. Overdose death rates. Accessed April 6, 2020 at: https://www.drugabuse.gov/related-topics/trends-statistics/overdose-death-rates
• 2 National Institute on Drug Abuse Missouri opioid summary. Accessed April 8, 2020 at: https://nida.nih.gov/drug-topics/opioids/opioid-summaries-by-state/missouri-opioid-involved-deaths-related-harms
• 3 Wilson N, Kariisa M, Seth P, Smith IV H, Davis NL. Drug and opioid-involved overdose deaths—United States, 2017-2018. MMWR Morb Mortal Wkly Rep 2020; 69 (11) 290-297
• 4 Bohnert ASB, Valenstein M, Bair MJ. et al. Association between opioid prescribing patterns and opioid overdose-related deaths. JAMA 2011; 305 (13) 1315-1321
• 5 Dowell D, Haegerich TM, Chou R. CDC guideline for prescribing opioids for chronic pain—United States, 2016. MMWR Recomm Rep 2016; 65 (01) 1-49
• 6 Nuckols TK, Anderson L, Popescu I. et al. Opioid prescribing: a systematic review and critical appraisal of guidelines for chronic pain. Ann Intern Med 2014; 160 (01) 38-47
• 7 Centers for Disease Control and Prevention.. Quality Improvement and Care Coordination: Implementing the CDC Guideline for Prescribing Opioids for Chronic Pain. Atlanta, GA: Division of Unintentinal Injury Prevention, National Center for Injury Prevention and Control; 2018
• 8 Starrels JL, Becker WC, Weiner MG, Li X, Heo M, Turner BJ. Low use of opioid risk reduction strategies in primary care even for high risk patients with chronic pain. J Gen Intern Med 2011; 26 (09) 958-964
• 9 Tournebize J, Gibaja V, Muszczak A, Kahn JP. Are physicians safely prescribing opioids for chronic noncancer pain? A systematic review of current evidence. Pain Pract 2016; 16 (03) 370-383
• 10 Losby JL, Hyatt JD, Kanter MH, Baldwin G, Matsuoka D. Safer and more appropriate opioid prescribing: a large healthcare system's comprehensive approach. J Eval Clin Pract 2017; 23 (06) 1173-1179
• 11 Huang KTL, Blazey-Martin D, Chandler D, Wurcel A, Gillis J, Tishler J. A multicomponent intervention to improve adherence to opioid prescribing and monitoring guidelines in primary care. J Opioid Manag 2019; 15 (06) 445-453
• 12 Dorflinger L, Moore B, Goulet J. et al. A partnered approach to opioid management, guideline concordant care and the stepped care model of pain management. J Gen Intern Med 2014; 29 (Suppl. 04) 870-876
• 13 Nielsen J. 10 Usability Heuristics for User Interface Design. Nielsen Norman Group.. Accessed May 17, 2020 at: https://www.nngroup.com/articles/ten-usability-heuristics/
• 14 Boeger MR. 7-Day Opiate Supply for Initial Prescriptions for Acute Pain. Missouri Bureau of Narcotics and Dangerous Drugs. Aug 29, 2018. Available at: https://health.mo.gov/safety/bndd/pdf/7-day-opiate-prescriptions.pdf
• 15 Opioid Oral Morphine Milligram Equivalent (MME) Conversion Factors. Centers for Medicare & Medicaid Services. 2020. HHS-0938-2020-F-5924. Available at: https://www.hhs.gov/guidance/document/opioid-oral-morphine-milligram-equivalent-mme-conversion-factors-0
• 16 Value Set Authority Center.. NLM Value Set Repository. Accessed April 15, 2020 at: https://vsac.nlm.nih.gov/
• 17 Linden A. Conducting interrupted time-series analysis for single- and multiple-group comparisons. Stata J 2015; 15 (02) 480-500
• 18 Lopez Bernal J, Cummins S, Gasparrini A. The use of controls in interrupted time series studies of public health interventions. Int J Epidemiol 2018; 47 (06) 2082-2093
• 19 Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf 2016; 25 (12) 986-992
• 20 Smart R, Pardo B, Davis CS. Systematic review of the emerging literature on the effectiveness of naloxone access laws in the United States. Addiction 2021; 116 (01) 6-17
• 21 Guy Jr GP, Haegerich TM, Evans ME, Losby JL, Young R, Jones CM. Vital signs: pharmacy-based naloxone dispensing—United States, 2012-2018. MMWR Morb Mortal Wkly Rep 2019; 68 (31) 679-686
• 22 Starrels JL, Becker WC, Alford DP, Kapoor A, Williams AR, Turner BJ. Systematic review: treatment agreements and urine drug testing to reduce opioid misuse in patients with chronic pain. Ann Intern Med 2010; 152 (11) 712-720
• 23 Chapman L, dela Cruz AM, Hutto J. Opioid treatment agreements: helpful or hurtful?. Am J Psychiatry Resid J 2016; 11 (09) 14-16
• 24 McGlothlin AE, Lewis RJ. Minimal clinically important difference: defining what really matters to patients. JAMA 2014; 312 (13) 1342-1343
• 25 Hawkins AT, Samuels LR. Use of confidence intervals in interpreting nonstatistically significant results. JAMA 2021; 326 (20) 2068-2069
• 26 Osheroff JA. ed. Improving Medication Use and Outcome with Clinical Decision Support: a Step-by-step Guide. Healthcare Information and Management Systems Society; 2009
• 27 Ratwani RM, Savage E, Will A. et al. A usability and safety analysis of electronic health records: a multi-center study. J Am Med Inform Assoc 2018; 25 (09) 1197-1201
• 28 Ratwani R, Fairbanks T, Savage E. et al. Mind the Gap. A systematic review to identify usability and safety challenges and practices during electronic health record implementation. Appl Clin Inform 2016; 7 (04) 1069-1087
• 29 Hettinger AZ, Melnick ER, Ratwani RM. Advancing electronic health record vendor usability maturity: progress and next steps. J Am Med Inform Assoc 2021; 28 (05) 1029-1031
• 30 Monica K. Cerner Launching EHR-Integrated Opioid Toolkit for Safe Prescribing. Xtelligent Healthcare Media, LLC; 2021 Accessed August 1, 2021 at: https://ehrintelligence.com/news/cerner-launching-ehr-integrated-opioid-toolkit-for-safe-prescribing
• 31 Ross C. Epic's AI algorithms, shielded from scrutiny by a corporate firewall, are delivering inaccurate information on seriously ill patients. STAT.. 2021 Accessed August 1, 2021 at: https://www.statnews.com/2021/07/26/epic-hospital-algorithms-sepsis-investigation/
• 32 Wong A, Otles E, Donnelly JP. et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med 2021; 181 (08) 1065-1070
• 33 Coiera E, Tong HL. Replication studies in the clinical decision support literature-frequency, fidelity, and impact. J Am Med Inform Assoc 2021; 28 (09) 1815-1825
• 34 Smith LB, Golberstein E, Anderson K. et al. The association of EHR drug safety alerts and co-prescribing of opioids and benzodiazepines. J Gen Intern Med 2019; 34 (08) 1403-1405
• 35 Price-Haywood EG, Burton J, Burstain T. et al. Clinical effectiveness of decision support for prescribing opioids for chronic noncancer pain: a prospective cohort study. Value Health 2020; 23 (02) 157-163
• 36 Anderson DR, Zlateva I, Coman EN, Khatri K, Tian T, Kerns RD. Improving pain care through implementation of the stepped care model at a multisite community health center. J Pain Res 2016; 9: 1021-1029
• 37 Gregory ME, Russo E, Singh H. Electronic health record alert-related workload as a predictor of burnout in primary care providers. Appl Clin Inform 2017; 8 (03) 686-697
• 38 Melnick ER, Dyrbye LN, Sinsky CA. et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc 2020; 95 (03) 476-487
• 39 Committee on Patient Safety and Health Information Technology; Institute of Medicine.. Health IT and Patient Safety: Building Safer Systems for Better Care. Washington, DC: National Academies Press; 2011
• 40 Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA 2018; 319 (12) 1276-1278
• 41 Genco EK, Forster JE, Flaten H. et al. Clinically inconsequential alerts: the characteristics of opioid drug alerts and their utility in preventing adverse drug events in the emergency department. Ann Emerg Med 2016; 67 (02) 240-248