Keywords
computerized provider order entry system - clinical decision support - adverse drug event - patient safety - intensive care units
Background and Significance
Background and Significance
Medication errors are common in patient care and may lead to adverse drug events (ADEs) and significant harm.[1] Medication clinical decision support (CDS) systems have shown promise in reducing up to 81% of medication errors.[2] These errors could be prevented if adherence to such alerts and suggestions were higher, as one study showed that a fully implemented CDS system could identify 89% of existing medication errors and prevent 23% from happening.[3] One population at high risk for ADEs is patients admitted to the intensive care unit (ICU), due to the increased hospital length of stay and the higher number of medications administered to these patients.[4]
[5]
[6] Given the risk of harm from inefficient CDS systems, patient outcomes may be substantially improved by paring down clinically irrelevant alerts. However, many barriers exist to complete adherence and decision support is often not clinically relevant.
In most systems, the vast majority of these CDS alerts are overridden, often appropriately, likely due to alert fatigue and poor alert design.[7]
[8] One study reported the overall override rate of 53% and the appropriateness rate of these overrides of 53%.[8] Other studies have found even higher override rates.[9]
[10] With the persistence of alert fatigue, these large numbers of excess alerts open up the possibility of missing critical alerts that will lead to patient harm.[9]
[11]
[12]
[13]
[14] Therefore, it is critical to develop better systems for identifying excess alerts that can be suppressed.
Effective means of paring down clinically irrelevant alerts include tiering drug–drug interactions (DDIs) by potential severity and changing the alerting thresholds for doses of medications, which have resulted in increased provider compliance with these alerts (i.e., acceptance).[15]
[16] However, these studies only evaluated a single type of alert. Another method that is not well-studied is to evaluate the provider's reason for overriding the alert, which can be evaluated for multiple alert types. Data suggest that override reasons provided on order entry may not truly reflect the intention of the provider. One mechanism to improve the provided override reason is to customize the potential reasons that are allowed, which increased the rate of appropriate responses by approximately 12%.[17]
[18]
However, there is much evidence that providers may not act as expected due to the potential lack of accounting for factors such as cognitive and human factors design considerations.[9]
[19] One of the override reasons that is provided by our electronic health record (EHR) is “Inaccurate warning,” which if used correctly, may help us to identify clinically irrelevant alerts. Therefore, we aimed to understand the behavior of providers using this reason to determine if evaluating these overrides would provide us opportunities to improve the underlying knowledge base.
Objective
This article aims to understand when providers use the override reason “Inaccurate warning,” and if evaluation of the appropriateness of these overrides can identify clinically irrelevant alerts.
Materials and Methods
We performed this study in a large urban academic medical center that uses a leading vendor EHR system in the United States.[20] Alert logic was sourced from First Databank (South San Francisco, California, United States). The analyzed alerts were from a convenience sample of orders generated between June 15th and November 15th, 2016, and occurred while the patient was in one of the six hospital ICUs (two medical, two neurology, and two surgical). The ICU setting was chosen due to the higher risk for ADEs in this patient population, compared with ward patients. The data were obtained from the EHR's data warehouse. When overriding an alert at our institution, providers may override without explanation or choose one of six reasons for their action, depending on the type of alert. The options for overrides were coded and provided by our EHR, and included “Benefit outweighs risk,” “Does not apply to patient,” “Patient tolerated before,” “Per protocol,” “Will monitor,” or “Inaccurate warning.” If a coded reason was chosen for the override, the provider was not able to add comments regarding their override. The provider (anesthesiologist, fellow, nurse practitioner, physician, physician assistant, resident) had to have overridden the alert using the reason “Inaccurate warning,” and the alert must have been one of three high-significance alert types, selected because of their frequency and potential risks for an ICU population: drug-allergy, DDI, and dose. Dose alerts were further analyzed according to the following subtypes: overdose, renal, and weight. These subtypes were chosen because of the importance of each of these characteristics in the ICU (e.g., body weight extremes of patients, rate of acute kidney injury in this population).[21] There was no change in alert settings during this study period. We evaluated the frequency that “Inaccurate warning” was used and how often providers used this override reason.
Appropriateness Evaluation
Appropriateness of overrides was determined by thorough chart reviews completed by two independent reviewers using predetermined criteria. A third reviewer (physician with expertise in medication safety) was used if consensus could not be reached. The two reviewers were a clinical pharmacist and a research assistant with interest in pursuing a career in medicine. The reviewers underwent standardized training through Brigham and Women's Center for Patient Safety Research and Practice, which has been used in previous studies.[22]
[23]
[24] The first two reviewers agreed 94% of the time, with a κ-statistic of 0.87 (95% confidence interval [CI], 0.84–0.90), showing significant agreement. Generally, if the alert was relevant, accurate, and had the possibility to harm the patient, the override was considered inappropriate. For example, for a drug-allergy alert, the override was considered inappropriate if the patient had the documented allergy, their reaction was immune-mediated (e.g., anaphylaxis from morphine), the medication had not been tolerated since the recorded reaction, and they were not going through a desensitization protocol. For a DDI, the override was considered inappropriate if the interaction put the patient at risk for harm associated with the DDI, despite how rare the condition may be. For renal alerts, the overrides were considered inappropriate if the patient did have renal insufficiency and the dose was not adjusted for their level of impairment. Our criteria included the renal dosing guidelines at our institution. Overdose overrides were considered inappropriate if the dose was not supported by published literature, even when factoring in a patient's condition (e.g., weight). Weight overrides were considered inappropriate if the medication had a weight-based dosing mechanism that was either not used or used incorrectly. Medications that were continued from home were determined to be inappropriate if there was evidence of a documented potential adverse reaction to the medication during hospital admission.
Results
During the period examined, there were 26,501 overridden alerts in the ICUs, 268 of which were documented with the reason “Inaccurate warning” (1.0%) ([Fig. 1]). Ninety-three (35%) of these alerts were part of the three high-significance alert types (dose: n = 30; drug-allergy: n = 48; DDI: n = 15). These 93 overrides represented 0.4% of dose alerts, 3.1% of drug-allergy alerts, and 0.4% of DDI alerts that occurred in the ICUs during the studied time period. A total of 22 providers used this override reason (attending: n = 1; fellow: n = 2; resident: n = 19), with the five using this override reason most frequently also using many other override reasons. The attending used “Inaccurate warning” for one dose alert, while the two fellows used it for three drug-allergy alerts. The median number of times that “Inaccurate warning” was used for the residents was 2 (interquartile range [IQR], 1–5), with drug-allergy alerts the most commonly overridden (n = 45, 51%). The top user of this reason (resident, n = 24) only used it for 39% of his/her overrides. The total number of providers who overrode alerts in the ICU during the time period was 493, so only 4.5% used the reason “Inaccurate warning” for even one override. Overall, the override appropriateness rate was 66% for the three high-significance alert types. [Fig. 2] illustrates the rates of override appropriateness by alert type and subtype ([Fig. 2]).
Fig. 1 Study flow diagram.
Fig. 2 Rate of override appropriateness by alert type and subtype.
Evaluation of Dose Alerts
[Table 1] illustrates the appropriate and inappropriate overrides of the dose alert subtypes. The appropriate renal replacement therapy doses included appropriate doses of meropenem or levofloxacin during continuous venovenous hemofiltration therapy, and appropriately administered calcium gluconate, furosemide, and potassium chloride in patients receiving hemodialysis.
Table 1
Classification and justification of dose overrides by subtype
|
Alert type
|
Subtype
|
Appropriate “Inaccurate warning” overrides
|
Inappropriate “Inaccurate warning” overrides
|
|
Dose
(n = 30)
|
Renal
(n = 13)
|
• Appropriate renal doses (n = 7, 54%)
• Patients without renal insufficiency (n = 4, 31%)
|
• Inappropriate doses for patients with renal insufficiency (piperacillin–tazobactam) (n = 2, 15%)
|
|
Weight
(n = 4)
|
• Medications not requiring weight-based dosing (magnesium sulfate, ondansetron) (n = 2, 50%)
|
• Medications incorrectly dosed by documented patient weight (enoxaparin, vancomycin) (n = 2, 50%)
|
|
Overdose
(n = 13)
|
• Correct doses of potassium chloride for replacement (n = 3, 23%)
• Correct doses of other medications (antibiotics, famotidine, insulin for hyperkalemia, methadone) (n = 5, 39%)
|
• Inappropriate doses of home medications (n = 2, 15%)
• Inappropriate doses of medications ordered inpatient (oxycodone, sodium bicarbonate) (n = 2, 15%)
• Inappropriate duration of therapy for antibiotic (n = 1, 8%)
|
Evaluation of Drug-Allergy Alerts
[Table 2] illustrates the appropriate and inappropriate overrides of the drug-allergy alerts. Twenty-two unique allergens were included in these alerts. These incorrect exact ingredient matches were as follows: normal saline as the exact ingredient match between the allergen (heparin or meperidine) and the ordered medication (normal saline with potassium chloride or calcium gluconate, respectively), an allergy to ipratropium alerting on an order for albuterol (the allergen was incorrectly entered as the combined ipratropium bromide–albuterol sulfate nebulizer), or an anaphylactic reaction to seafood listed as an allergy to iodine that then alerted on an order for amiodarone. Despite these many appropriate overrides, only one (2.1%) of these allergies were modified during the admission to the ICU (ingredient match between heparin and normal saline with potassium chloride, where the normal saline portion of the allergy was removed).
Table 2
Classification and justification of drug-allergy overrides
|
Alert type
|
Appropriate “Inaccurate warning” overrides
|
Inappropriate “Inaccurate warning” overrides
|
|
Drug-allergy
(n = 48)
|
• Previously tolerated the medication (n = 30, 63%)
• Intolerances listed as allergies (e.g., gastrointestinal upset) (n = 5, 10%)
• Inaccurate ingredient matches (n = 5, 10%)
|
• Immune-mediated responses noted, and had not subsequently tolerated the medication (n = 8, 17%)
|
Evaluation of Drug–Drug Interaction Alerts
[Table 3] illustrates the appropriate and inappropriate overrides of the DDI alerts. All of the 15 DDI overrides were inappropriate. Ten different medication combinations were alerted on (amiodarone–atorvastatin, n = 1; amiodarone–levofloxacin, n = 2; amiodarone–quetiapine, n = 1; escitalopram–levofloxacin, n = 1; fluconazole–levofloxacin, n = 2; haloperidol–octreotide, n = 3; ondansetron–citalopram, n = 1; ondansetron–levofloxacin, n = 1; ondansetron–methadone, n = 2; ondansetron–trazodone, n = 1).
Table 3
Classification of drug–drug interaction overrides
|
Alert type
|
Appropriate “Inaccurate warning” overrides
|
Inappropriate “Inaccurate warning” overrides
|
|
Drug–Drug
|
N/A
|
• Interactions where risk was present (n = 15, 100%)
|
Discussion
Usage of the “Inaccurate warning” reason was infrequent (1.0% of all overrides). Even the most frequent users of this reason used it for less than half their overrides, suggesting a purposeful intent behind the selection, rather than using it as a default reason to override. When used, providers were often correct in overriding (66%). Auditing of this reason represents a rich resource for improving CDS, which could in turn reduce alert fatigue. The overall appropriateness rate and the override rates for dose and drug-allergy alerts were consistent with prior studies.[8] Each specific alert type revealed something different about excess alerts.
Discussion of Dose Alerts
The different subtypes of dose alerts each revealed something different about how to reduce the excess alerts. Like drug-allergy alerts and DDIs, dose alerts are known to have large proportions of overridden alerts, only a small fraction of which (0.4%) were captured by the “Inaccurate warning” reason. The high override appropriateness rate of these alerts (70%), and the justifications for their appropriateness, shows how excess alerts can be reduced. The renal overrides show that a better mechanism for identifying patients without renal injury and more accurate dosing criteria in patients who have renal injury, are needed. Similarly, the CDS system could better detect patient treatments, such as hemodialysis or continuous venovenous hemofiltration therapy, and account for altered dosing in those situations. The overridden weight alerts show that changing the alerting criteria to only warn on protocols actually requiring weight, and not alerting on nonweight-based protocols, could help reduce excess alerts. The overdose alerts show that improving the dose thresholds for various medications, especially electrolyte replacement protocols, could reduce many unhelpful alerts. Changing the alerting mechanisms and rectifying the alerting knowledge base could reduce excess alerts, enhancing provider trust in the alerts they receive. Future studies should expand this analysis and attempt to look at all of the dosing alert criteria, as dose alerts account for the majority of errors and ADEs, and are among the most straightforward alert criteria to analyze.[25]
[26]
Discussion of Drug-Allergy Alerts
Prior research has shown that for drug-allergy alerts, despite their high override rate (∼80%), almost all overrides are appropriate and do not result in harm.[13] This was consistent with the study data, which showed that the majority of drug-allergy alert overrides using “Inaccurate warning” were appropriate (83%). This reinforces the need to reduce excess drug-allergy alerts. The appropriate overrides collected here suggest that patient allergy list management requires improvement, leading to more tailored, accurate alerts. The persistence of listed allergies to medications that have been tolerated without reaction, are incorrectly classified intolerances, or are inaccurate ingredient matches is not a new phenomenon, but points to both user and systemic errors. Prior studies have documented the rate of providers cancelling alerted-upon allergies as high as 17%, though only 2% of inaccurate allergies were cancelled in this study.[13] Even though providers identify spurious allergies, they do not alter the lists.
The prior homegrown EHR and CDS system at the hospital system allowed providers to cancel allergies while ordering medications, such that if they overrode an allergy with the reason “patient does not have this allergy, will d/c preexisting allergy,” they were prompted with an option to discontinue the allergy at that point in time. This might solve the above issues if it were implemented in the vendor system, as providers identifying an incorrect ingredient match or an inappropriately listed intolerance could edit the allergy list while they are actively identifying the problem. However, if override reasons are to be used to identify these alerts, it may be worthwhile for future studies to analyze other reasons, such as “Patient tolerated before” or “Will monitor,” as these may capture more of the excess alerts. Prior studies of systems with free-text override reasons have shown for drug-allergy alerts the reasons most closely corresponding to “Patient tolerated before” were the most popular, with “Aware” as the second most popular, so the appropriateness of overrides with a similar reasons should be analyzed.[9]
[23]
Discussion of Drug–Drug Interaction Alerts
The literature points to DDIs having the highest override rates (∼87%) and many unhelpful alerts (eight out of nine).[7]
[10]
[12] However, the “Inaccurate warning” reason failed to capture any of these alerts. Instead, providers used this reason to override alerts that were in fact accurate and applicable, and were warning them about possible patient harms. It is revealing that providers did not use this reason to label any of the unhelpful excess alerts. As our criteria were based on the potential of harm (and not risk) and that the alert was “inaccurate,” there would be few examples of an “inaccurate” DDI. This may include an alert on the administration of intravenous ciprofloxacin and an oral antacid (DDI is reduced absorption of ciprofloxacin), which would be an inaccurate alert, as this is only a concern when both medications are administered orally. Our evaluation could help address these alerts that are always clinically irrelevant. As most of the DDIs in our study were associated with QT-prolongation, other clinical factors such as multiple QT-prolonging medications and electrolyte status may affect a patient's risk. Providers may believe that these alerts are inaccurate due to the close monitoring of patients provided in the ICU but there is risk involved. Though our sample size was modest, auditing of this class was not helpful in finding alerts which could be eliminated.
Future Directions
Although there is no established threshold of how often a CDS alert should be accepted (i.e., not overridden), experts have suggested a rate of at least 60% based on opinion.[27] Education of providers that “Inaccurate warning” is a continuously monitored override reason may help providers to aid in the identification of certain alerts as clinically irrelevant. This may increase buy-in to systems that is required for clinicians to trust the technology that is used to improve patient outcomes.[28]
[29]
[30] Although there are certainly concerns of how much effort would be needed to evaluate these alerts (which was not determined in this study), we believe that this can help to identify malfunctioning or alerts that are always clinically irrelevant (e.g., ciprofloxacin and antacid example from the “Discussion of Drug–Drug Interaction Alerts” section). A more real-time evaluation (e.g., electronic means through a reporting system, medication safety professional evaluating these alerts) would help identify the reason a provider used the “Inaccurate warning” override reason. This may identify a truly inaccurate alert or if provider education regarding the clinical significance of the alert is needed.
Limitations
Our study has several limitations. First, this study was performed at a single institution and findings from this study may not be applicable to other institutions. Medical residents likely accounted for the vast majority of the overrides as they are the ones who interact with medication ordering the most of any provider, given that we are an academic medical center. Our institution utilizes an EHR and knowledge base that may not be used by other institutions. Alert logic from knowledge bases are based on set criteria but may be tailored to institution-specific needs. Given that the EHR we use is among the most commonly adopted, we believe that our situation applies to a significant number of other institutions with a similar commercial EHR. Second, we may not have all the necessary information to identify why a provider believed the alert to be inaccurate. The intent of this study was to identify inaccurate alerts (i.e., alerts that had no clinical relevance to the patient). We may not have exhaustively evaluated factors that the provider may have considered in determining the decision to override a CDS alert, which could have been identified by an interview. We were limited given the retrospective nature of this study in efforts to interview providers at the time of override. Nevertheless, we made considerable efforts to evaluate the appropriateness of overrides and subsequent ADEs on a case-by-case basis, formulation of criteria using a multidisciplinary expert team, and the use of independent adjudicators. Third, we did not evaluate if any of the overrides resulted in patient harm. Identification of patient harm would help illustrate the outcomes of these overrides and their impact on patient morbidity and mortality. Finally, the findings of our study may not be able to extrapolated to settings outside the ICU. Although we did target the ICU patient population because they are especially vulnerable to ADEs, none of the alerts that we evaluated are truly ICU-specific. The method that we used to evaluate overrides can also be used outside the ICU and at institutions that have a similar override reason.
Conclusion
Overall, the “Inaccurate warning” reason only captured a small number of the excess alerts known to exist, approximately1.0% of all overrides during the study time period. However, its use revealed important ways to reduce excess alerts. Notably, the problems identified with these alerts are consistent with those identified over a decade ago, showing the importance of finding new ways to identify and remove excess alerts.[1]
[9]
[13]
[25]
[31] Future studies should analyze other override reasons to determine whether they can better identify these alerts, and should focus on improving the CDS systems to make the given alerts more targeted and helpful—to relieve provider fatigue and to improve patient care and safety.
Clinical Relevance Statement
Clinical Relevance Statement
In this study, we quantified provider behavior around the use of the “Inaccurate warning” override reason, and analyzed the justifications for why such overrides were inappropriate or not. The override data suggested many specific improvements of CDS systems that could reduce the number of unhelpful alerts shown to providers. If such analysis were extrapolated and used as a way of monitoring alert overrides, alert fatigue may be reduced, providers may override fewer relevant alerts, and patient safety could improve.
Multiple Choice Question
What is one significant barrier to the effective use of a CDS system?
-
Creating well-designed alerts
-
Alert fatigue
-
Researching systems to reduce alert volume
-
Provider clinical knowledge
Correct Answer: The correct answer is option b, alert fatigue. Alert fatigue causes providers to ignore large proportions of CDS alerts, including those that may have been relevant and could have prevented patient harms. For example, providers override around 53% of all alerts, at an appropriateness rate of only approximately 53%.[5] It has been shown that regardless of clinical value, provider acceptance of alerts decreases 30% with each additional alert shown.[8]
