Determining Inappropriate Medication Alerts from “Inaccurate Warning” Overrides in the Intensive Care Unit

Abstract

Objective This article aims to understand provider behavior around the use of the override reason “Inaccurate warning,” specifically whether it is an effective way of identifying unhelpful medication alerts.

Materials and Methods We analyzed alert overrides that occurred in the intensive care units (ICUs) of a major academic medical center between June and November 2016, focused on the following high-significance alert types: dose, drug-allergy alerts, and drug–drug interactions (DDI). Override appropriateness was analyzed by two independent reviewers using predetermined criteria.

Results A total of 268 of 26,501 ICU overrides (1.0%) used the reason “Inaccurate warning,” with 93 of these overrides associated with our included alert types. Sixty-one of these overrides (66%) were identified to be appropriate. Twenty-one of 30 (70%) dose alert overrides were appropriate. Forty of 48 drug-allergy alert overrides (83%) were appropriate, for reasons ranging from prior tolerance (n = 30) to inaccurate ingredient matches (n = 5). None of the 15 DDI overrides were appropriate.

Conclusion The “Inaccurate warning” reason was selectively used by a small proportion of providers and overrides using this reason identified important opportunities to reduce excess alerts. Potential opportunities include improved evaluation of dosing mechanisms based on patient characteristics, inclusion of institutional dosing protocols to alert logic, and evaluation of a patient's prior tolerance to a medication that they have a documented allergy for. This resource is not yet routinely used for alert tailoring at our institution but may prove to be a valuable resource to evaluate available alerts.

Keywords

computerized provider order entry system - clinical decision support - adverse drug event - patient safety - intensive care units

Background and Significance

Medication errors are common in patient care and may lead to adverse drug events (ADEs) and significant harm.[1] Medication clinical decision support (CDS) systems have shown promise in reducing up to 81% of medication errors.[2] These errors could be prevented if adherence to such alerts and suggestions were higher, as one study showed that a fully implemented CDS system could identify 89% of existing medication errors and prevent 23% from happening.[3] One population at high risk for ADEs is patients admitted to the intensive care unit (ICU), due to the increased hospital length of stay and the higher number of medications administered to these patients.[4] [5] [6] Given the risk of harm from inefficient CDS systems, patient outcomes may be substantially improved by paring down clinically irrelevant alerts. However, many barriers exist to complete adherence and decision support is often not clinically relevant.

In most systems, the vast majority of these CDS alerts are overridden, often appropriately, likely due to alert fatigue and poor alert design.[7] [8] One study reported the overall override rate of 53% and the appropriateness rate of these overrides of 53%.[8] Other studies have found even higher override rates.[9] [10] With the persistence of alert fatigue, these large numbers of excess alerts open up the possibility of missing critical alerts that will lead to patient harm.[9] [11] [12] [13] [14] Therefore, it is critical to develop better systems for identifying excess alerts that can be suppressed.

Effective means of paring down clinically irrelevant alerts include tiering drug–drug interactions (DDIs) by potential severity and changing the alerting thresholds for doses of medications, which have resulted in increased provider compliance with these alerts (i.e., acceptance).[15] [16] However, these studies only evaluated a single type of alert. Another method that is not well-studied is to evaluate the provider's reason for overriding the alert, which can be evaluated for multiple alert types. Data suggest that override reasons provided on order entry may not truly reflect the intention of the provider. One mechanism to improve the provided override reason is to customize the potential reasons that are allowed, which increased the rate of appropriate responses by approximately 12%.[17] [18]

However, there is much evidence that providers may not act as expected due to the potential lack of accounting for factors such as cognitive and human factors design considerations.[9] [19] One of the override reasons that is provided by our electronic health record (EHR) is “Inaccurate warning,” which if used correctly, may help us to identify clinically irrelevant alerts. Therefore, we aimed to understand the behavior of providers using this reason to determine if evaluating these overrides would provide us opportunities to improve the underlying knowledge base.

Objective

This article aims to understand when providers use the override reason “Inaccurate warning,” and if evaluation of the appropriateness of these overrides can identify clinically irrelevant alerts.

Materials and Methods

We performed this study in a large urban academic medical center that uses a leading vendor EHR system in the United States.[20] Alert logic was sourced from First Databank (South San Francisco, California, United States). The analyzed alerts were from a convenience sample of orders generated between June 15th and November 15th, 2016, and occurred while the patient was in one of the six hospital ICUs (two medical, two neurology, and two surgical). The ICU setting was chosen due to the higher risk for ADEs in this patient population, compared with ward patients. The data were obtained from the EHR's data warehouse. When overriding an alert at our institution, providers may override without explanation or choose one of six reasons for their action, depending on the type of alert. The options for overrides were coded and provided by our EHR, and included “Benefit outweighs risk,” “Does not apply to patient,” “Patient tolerated before,” “Per protocol,” “Will monitor,” or “Inaccurate warning.” If a coded reason was chosen for the override, the provider was not able to add comments regarding their override. The provider (anesthesiologist, fellow, nurse practitioner, physician, physician assistant, resident) had to have overridden the alert using the reason “Inaccurate warning,” and the alert must have been one of three high-significance alert types, selected because of their frequency and potential risks for an ICU population: drug-allergy, DDI, and dose. Dose alerts were further analyzed according to the following subtypes: overdose, renal, and weight. These subtypes were chosen because of the importance of each of these characteristics in the ICU (e.g., body weight extremes of patients, rate of acute kidney injury in this population).[21] There was no change in alert settings during this study period. We evaluated the frequency that “Inaccurate warning” was used and how often providers used this override reason.

Appropriateness Evaluation

Appropriateness of overrides was determined by thorough chart reviews completed by two independent reviewers using predetermined criteria. A third reviewer (physician with expertise in medication safety) was used if consensus could not be reached. The two reviewers were a clinical pharmacist and a research assistant with interest in pursuing a career in medicine. The reviewers underwent standardized training through Brigham and Women's Center for Patient Safety Research and Practice, which has been used in previous studies.[22] [23] [24] The first two reviewers agreed 94% of the time, with a κ-statistic of 0.87 (95% confidence interval [CI], 0.84–0.90), showing significant agreement. Generally, if the alert was relevant, accurate, and had the possibility to harm the patient, the override was considered inappropriate. For example, for a drug-allergy alert, the override was considered inappropriate if the patient had the documented allergy, their reaction was immune-mediated (e.g., anaphylaxis from morphine), the medication had not been tolerated since the recorded reaction, and they were not going through a desensitization protocol. For a DDI, the override was considered inappropriate if the interaction put the patient at risk for harm associated with the DDI, despite how rare the condition may be. For renal alerts, the overrides were considered inappropriate if the patient did have renal insufficiency and the dose was not adjusted for their level of impairment. Our criteria included the renal dosing guidelines at our institution. Overdose overrides were considered inappropriate if the dose was not supported by published literature, even when factoring in a patient's condition (e.g., weight). Weight overrides were considered inappropriate if the medication had a weight-based dosing mechanism that was either not used or used incorrectly. Medications that were continued from home were determined to be inappropriate if there was evidence of a documented potential adverse reaction to the medication during hospital admission.

Results

During the period examined, there were 26,501 overridden alerts in the ICUs, 268 of which were documented with the reason “Inaccurate warning” (1.0%) ([Fig. 1]). Ninety-three (35%) of these alerts were part of the three high-significance alert types (dose: n = 30; drug-allergy: n = 48; DDI: n = 15). These 93 overrides represented 0.4% of dose alerts, 3.1% of drug-allergy alerts, and 0.4% of DDI alerts that occurred in the ICUs during the studied time period. A total of 22 providers used this override reason (attending: n = 1; fellow: n = 2; resident: n = 19), with the five using this override reason most frequently also using many other override reasons. The attending used “Inaccurate warning” for one dose alert, while the two fellows used it for three drug-allergy alerts. The median number of times that “Inaccurate warning” was used for the residents was 2 (interquartile range [IQR], 1–5), with drug-allergy alerts the most commonly overridden (n = 45, 51%). The top user of this reason (resident, n = 24) only used it for 39% of his/her overrides. The total number of providers who overrode alerts in the ICU during the time period was 493, so only 4.5% used the reason “Inaccurate warning” for even one override. Overall, the override appropriateness rate was 66% for the three high-significance alert types. [Fig. 2] illustrates the rates of override appropriateness by alert type and subtype ([Fig. 2]).

Fig. 1 Study flow diagram.

Fig. 2 Rate of override appropriateness by alert type and subtype.

Evaluation of Dose Alerts

[Table 1] illustrates the appropriate and inappropriate overrides of the dose alert subtypes. The appropriate renal replacement therapy doses included appropriate doses of meropenem or levofloxacin during continuous venovenous hemofiltration therapy, and appropriately administered calcium gluconate, furosemide, and potassium chloride in patients receiving hemodialysis.

Table 1 Classification and justification of dose overrides by subtype

Alert type	Subtype	Appropriate “Inaccurate warning” overrides	Inappropriate “Inaccurate warning” overrides
Dose (n = 30)	Renal (n = 13)	• Appropriate renal doses (n = 7, 54%) • Patients without renal insufficiency (n = 4, 31%)	• Inappropriate doses for patients with renal insufficiency (piperacillin–tazobactam) (n = 2, 15%)
	Weight (n = 4)	• Medications not requiring weight-based dosing (magnesium sulfate, ondansetron) (n = 2, 50%)	• Medications incorrectly dosed by documented patient weight (enoxaparin, vancomycin) (n = 2, 50%)
	Overdose (n = 13)	• Correct doses of potassium chloride for replacement (n = 3, 23%) • Correct doses of other medications (antibiotics, famotidine, insulin for hyperkalemia, methadone) (n = 5, 39%)	• Inappropriate doses of home medications (n = 2, 15%) • Inappropriate doses of medications ordered inpatient (oxycodone, sodium bicarbonate) (n = 2, 15%) • Inappropriate duration of therapy for antibiotic (n = 1, 8%)

Evaluation of Drug-Allergy Alerts

[Table 2] illustrates the appropriate and inappropriate overrides of the drug-allergy alerts. Twenty-two unique allergens were included in these alerts. These incorrect exact ingredient matches were as follows: normal saline as the exact ingredient match between the allergen (heparin or meperidine) and the ordered medication (normal saline with potassium chloride or calcium gluconate, respectively), an allergy to ipratropium alerting on an order for albuterol (the allergen was incorrectly entered as the combined ipratropium bromide–albuterol sulfate nebulizer), or an anaphylactic reaction to seafood listed as an allergy to iodine that then alerted on an order for amiodarone. Despite these many appropriate overrides, only one (2.1%) of these allergies were modified during the admission to the ICU (ingredient match between heparin and normal saline with potassium chloride, where the normal saline portion of the allergy was removed).

Table 2 Classification and justification of drug-allergy overrides

Alert type	Appropriate “Inaccurate warning” overrides	Inappropriate “Inaccurate warning” overrides
Drug-allergy (n = 48)	• Previously tolerated the medication (n = 30, 63%) • Intolerances listed as allergies (e.g., gastrointestinal upset) (n = 5, 10%) • Inaccurate ingredient matches (n = 5, 10%)	• Immune-mediated responses noted, and had not subsequently tolerated the medication (n = 8, 17%)

Evaluation of Drug–Drug Interaction Alerts

[Table 3] illustrates the appropriate and inappropriate overrides of the DDI alerts. All of the 15 DDI overrides were inappropriate. Ten different medication combinations were alerted on (amiodarone–atorvastatin, n = 1; amiodarone–levofloxacin, n = 2; amiodarone–quetiapine, n = 1; escitalopram–levofloxacin, n = 1; fluconazole–levofloxacin, n = 2; haloperidol–octreotide, n = 3; ondansetron–citalopram, n = 1; ondansetron–levofloxacin, n = 1; ondansetron–methadone, n = 2; ondansetron–trazodone, n = 1).

Table 3 Classification of drug–drug interaction overrides

Alert type	Appropriate “Inaccurate warning” overrides	Inappropriate “Inaccurate warning” overrides
Drug–Drug	N/A	• Interactions where risk was present (n = 15, 100%)

Discussion

Usage of the “Inaccurate warning” reason was infrequent (1.0% of all overrides). Even the most frequent users of this reason used it for less than half their overrides, suggesting a purposeful intent behind the selection, rather than using it as a default reason to override. When used, providers were often correct in overriding (66%). Auditing of this reason represents a rich resource for improving CDS, which could in turn reduce alert fatigue. The overall appropriateness rate and the override rates for dose and drug-allergy alerts were consistent with prior studies.[8] Each specific alert type revealed something different about excess alerts.

Discussion of Dose Alerts

The different subtypes of dose alerts each revealed something different about how to reduce the excess alerts. Like drug-allergy alerts and DDIs, dose alerts are known to have large proportions of overridden alerts, only a small fraction of which (0.4%) were captured by the “Inaccurate warning” reason. The high override appropriateness rate of these alerts (70%), and the justifications for their appropriateness, shows how excess alerts can be reduced. The renal overrides show that a better mechanism for identifying patients without renal injury and more accurate dosing criteria in patients who have renal injury, are needed. Similarly, the CDS system could better detect patient treatments, such as hemodialysis or continuous venovenous hemofiltration therapy, and account for altered dosing in those situations. The overridden weight alerts show that changing the alerting criteria to only warn on protocols actually requiring weight, and not alerting on nonweight-based protocols, could help reduce excess alerts. The overdose alerts show that improving the dose thresholds for various medications, especially electrolyte replacement protocols, could reduce many unhelpful alerts. Changing the alerting mechanisms and rectifying the alerting knowledge base could reduce excess alerts, enhancing provider trust in the alerts they receive. Future studies should expand this analysis and attempt to look at all of the dosing alert criteria, as dose alerts account for the majority of errors and ADEs, and are among the most straightforward alert criteria to analyze.[25] [26]

Discussion of Drug-Allergy Alerts

Prior research has shown that for drug-allergy alerts, despite their high override rate (∼80%), almost all overrides are appropriate and do not result in harm.[13] This was consistent with the study data, which showed that the majority of drug-allergy alert overrides using “Inaccurate warning” were appropriate (83%). This reinforces the need to reduce excess drug-allergy alerts. The appropriate overrides collected here suggest that patient allergy list management requires improvement, leading to more tailored, accurate alerts. The persistence of listed allergies to medications that have been tolerated without reaction, are incorrectly classified intolerances, or are inaccurate ingredient matches is not a new phenomenon, but points to both user and systemic errors. Prior studies have documented the rate of providers cancelling alerted-upon allergies as high as 17%, though only 2% of inaccurate allergies were cancelled in this study.[13] Even though providers identify spurious allergies, they do not alter the lists.

The prior homegrown EHR and CDS system at the hospital system allowed providers to cancel allergies while ordering medications, such that if they overrode an allergy with the reason “patient does not have this allergy, will d/c preexisting allergy,” they were prompted with an option to discontinue the allergy at that point in time. This might solve the above issues if it were implemented in the vendor system, as providers identifying an incorrect ingredient match or an inappropriately listed intolerance could edit the allergy list while they are actively identifying the problem. However, if override reasons are to be used to identify these alerts, it may be worthwhile for future studies to analyze other reasons, such as “Patient tolerated before” or “Will monitor,” as these may capture more of the excess alerts. Prior studies of systems with free-text override reasons have shown for drug-allergy alerts the reasons most closely corresponding to “Patient tolerated before” were the most popular, with “Aware” as the second most popular, so the appropriateness of overrides with a similar reasons should be analyzed.[9] [23]

Discussion of Drug–Drug Interaction Alerts

The literature points to DDIs having the highest override rates (∼87%) and many unhelpful alerts (eight out of nine).[7] [10] [12] However, the “Inaccurate warning” reason failed to capture any of these alerts. Instead, providers used this reason to override alerts that were in fact accurate and applicable, and were warning them about possible patient harms. It is revealing that providers did not use this reason to label any of the unhelpful excess alerts. As our criteria were based on the potential of harm (and not risk) and that the alert was “inaccurate,” there would be few examples of an “inaccurate” DDI. This may include an alert on the administration of intravenous ciprofloxacin and an oral antacid (DDI is reduced absorption of ciprofloxacin), which would be an inaccurate alert, as this is only a concern when both medications are administered orally. Our evaluation could help address these alerts that are always clinically irrelevant. As most of the DDIs in our study were associated with QT-prolongation, other clinical factors such as multiple QT-prolonging medications and electrolyte status may affect a patient's risk. Providers may believe that these alerts are inaccurate due to the close monitoring of patients provided in the ICU but there is risk involved. Though our sample size was modest, auditing of this class was not helpful in finding alerts which could be eliminated.

Future Directions

Although there is no established threshold of how often a CDS alert should be accepted (i.e., not overridden), experts have suggested a rate of at least 60% based on opinion.[27] Education of providers that “Inaccurate warning” is a continuously monitored override reason may help providers to aid in the identification of certain alerts as clinically irrelevant. This may increase buy-in to systems that is required for clinicians to trust the technology that is used to improve patient outcomes.[28] [29] [30] Although there are certainly concerns of how much effort would be needed to evaluate these alerts (which was not determined in this study), we believe that this can help to identify malfunctioning or alerts that are always clinically irrelevant (e.g., ciprofloxacin and antacid example from the “Discussion of Drug–Drug Interaction Alerts” section). A more real-time evaluation (e.g., electronic means through a reporting system, medication safety professional evaluating these alerts) would help identify the reason a provider used the “Inaccurate warning” override reason. This may identify a truly inaccurate alert or if provider education regarding the clinical significance of the alert is needed.

Limitations

Our study has several limitations. First, this study was performed at a single institution and findings from this study may not be applicable to other institutions. Medical residents likely accounted for the vast majority of the overrides as they are the ones who interact with medication ordering the most of any provider, given that we are an academic medical center. Our institution utilizes an EHR and knowledge base that may not be used by other institutions. Alert logic from knowledge bases are based on set criteria but may be tailored to institution-specific needs. Given that the EHR we use is among the most commonly adopted, we believe that our situation applies to a significant number of other institutions with a similar commercial EHR. Second, we may not have all the necessary information to identify why a provider believed the alert to be inaccurate. The intent of this study was to identify inaccurate alerts (i.e., alerts that had no clinical relevance to the patient). We may not have exhaustively evaluated factors that the provider may have considered in determining the decision to override a CDS alert, which could have been identified by an interview. We were limited given the retrospective nature of this study in efforts to interview providers at the time of override. Nevertheless, we made considerable efforts to evaluate the appropriateness of overrides and subsequent ADEs on a case-by-case basis, formulation of criteria using a multidisciplinary expert team, and the use of independent adjudicators. Third, we did not evaluate if any of the overrides resulted in patient harm. Identification of patient harm would help illustrate the outcomes of these overrides and their impact on patient morbidity and mortality. Finally, the findings of our study may not be able to extrapolated to settings outside the ICU. Although we did target the ICU patient population because they are especially vulnerable to ADEs, none of the alerts that we evaluated are truly ICU-specific. The method that we used to evaluate overrides can also be used outside the ICU and at institutions that have a similar override reason.

Conclusion

Overall, the “Inaccurate warning” reason only captured a small number of the excess alerts known to exist, approximately1.0% of all overrides during the study time period. However, its use revealed important ways to reduce excess alerts. Notably, the problems identified with these alerts are consistent with those identified over a decade ago, showing the importance of finding new ways to identify and remove excess alerts.[1] [9] [13] [25] [31] Future studies should analyze other override reasons to determine whether they can better identify these alerts, and should focus on improving the CDS systems to make the given alerts more targeted and helpful—to relieve provider fatigue and to improve patient care and safety.

Clinical Relevance Statement

In this study, we quantified provider behavior around the use of the “Inaccurate warning” override reason, and analyzed the justifications for why such overrides were inappropriate or not. The override data suggested many specific improvements of CDS systems that could reduce the number of unhelpful alerts shown to providers. If such analysis were extrapolated and used as a way of monitoring alert overrides, alert fatigue may be reduced, providers may override fewer relevant alerts, and patient safety could improve.

Multiple Choice Question

What is one significant barrier to the effective use of a CDS system?

• Creating well-designed alerts

• Alert fatigue

• Researching systems to reduce alert volume

• Provider clinical knowledge

  Correct Answer: The correct answer is option b, alert fatigue. Alert fatigue causes providers to ignore large proportions of CDS alerts, including those that may have been relevant and could have prevented patient harms. For example, providers override around 53% of all alerts, at an appropriateness rate of only approximately 53%.[5] It has been shown that regardless of clinical value, provider acceptance of alerts decreases 30% with each additional alert shown.[8]

Conflict of Interest

Dr. Bates reported receiving equity from Intensix, which makes software to support clinical decision-making in intensive care; being named as coinventor on Patent Number 6029138 held by Brigham and Women's Hospital on the use of decision support software for medical management, licensed to the Medicalis, and holding a minority equity position in Medicalis, which develops Web-based decision support for radiology test ordering; consulting for Early Sense, which makes patient safety monitoring systems; receiving equity and cash compensation from QPID, a company focused on intelligence systems for electronic health records; receiving cash compensation from CDI (Negev), which is a not-for-profit incubator for health IT start-ups; receiving equity from Enelgy, which makes software to support evidence-based clinical decisions, from Ethosmart, which makes software to help patients with chronic diseases, and from MDClone, which takes clinical data and produces deidentified versions of it. The remaining authors have disclosed that they do not have any conflict of interest.

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by Partners HealthCare Institutional Review Board.

• References

• 1 Lesar TS, Briceland L, Stein DS. Factors related to errors in medication prescribing. JAMA 1997; 277 (04) 312-317
• 2 Bates DW, Teich JM, Lee J. , et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999; 6 (04) 313-321
• 3 Bates DW, O'Neil AC, Boyle D. , et al. Potential identifiability and preventability of adverse events using information systems. J Am Med Inform Assoc 1994; 1 (05) 404-411
• 4 Bates DW, Cullen DJ, Laird N. , et al; ADE Prevention Study Group. Incidence of adverse drug events and potential adverse drug events. Implications for prevention. JAMA 1995; 274 (01) 29-34
• 5 Cullen DJ, Sweitzer BJ, Bates DW, Burdick E, Edmondson A, Leape LL. Preventable adverse drug events in hospitalized patients: a comparative study of intensive care and general care units. Crit Care Med 1997; 25 (08) 1289-1297
• 6 Morimoto T, Sakuma M, Matsui K. , et al. Incidence of adverse drug events and medication errors in Japan: the JADE study. J Gen Intern Med 2011; 26 (02) 148-153
• 7 Lin CP, Payne TH, Nichol WP, Hoey PJ, Anderson CL, Gennari JH. Evaluating clinical decision support systems: monitoring CPOE order check override rates in the Department of Veterans Affairs' Computerized Patient Record System. J Am Med Inform Assoc 2008; 15 (05) 620-626
• 8 Nanji KC, Slight SP, Seger DL. , et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc 2014; 21 (03) 487-491
• 9 Kuperman GJ, Bobb A, Payne TH. , et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc 2007; 14 (01) 29-40
• 10 Payne TH, Nichol WP, Hoey P, Savarino J. Characteristics and override rates of order checks in a practitioner order entry system. Proc AMIA Symp 2002; 2002: 602-606
• 11 Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. ; with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17 (01) 36
• 12 Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003; 163 (21) 2625-2631
• 13 Hsieh TC, Kuperman GJ, Jaggi T. , et al. Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system. J Am Med Inform Assoc 2004; 11 (06) 482-491
• 14 Abookire SA, Teich JM, Sandige H. , et al. Improving allergy alerting in a computerized physician order entry system. Proc AMIA Symp 2000; 2000: 2-6
• 15 Paterno MD, Maviglia SM, Gorman PN. , et al. Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc 2009; 16 (01) 40-46
• 16 Czock D, Konias M, Seidling HM. , et al. Tailoring of alerts substantially reduces the alert burden in computerized clinical decision support for drugs that should be avoided in patients with renal disease. J Am Med Inform Assoc 2015; 22 (04) 881-887
• 17 Cho I, Slight SP, Nanji KC. , et al. Understanding physicians' behavior toward alerts about nephrotoxic medications in outpatients: a cross-sectional analysis. BMC Nephrol 2014; 15: 200
• 18 Dekarske BM, Zimmerman CR, Chang R, Grant PJ, Chaffee BW. Increased appropriateness of customized alert acknowledgement reasons for overridden medication alerts in a computerized provider order entry system. Int J Med Inform 2015; 84 (12) 1085-1093
• 19 Bloomrosen M, Starren J, Lorenzi NM, Ash JS, Patel VL, Shortliffe EH. Anticipating and addressing the unintended consequences of health IT and policy: a report from the AMIA 2009 Health Policy Meeting. J Am Med Inform Assoc 2011; 18 (01) 82-90
• 20 Office of the National Coordinator for Health Information Technology. Certified Health IT Developers and Editions Reported by Hospitals Participating in the Medicare EHR Incentive Program. Washington, DC: HealthIT.gov; c2017. Available at: https://dashboard.healthit.gov/quickstats/pages/FIG-Vendors-of-EHRs-to-Participating-Hospitals.php . Accessed January 15, 2018
• 21 Hoste EA, Bagshaw SM, Bellomo R. , et al. Epidemiology of acute kidney injury in critically ill patients: the multinational AKI-EPI study. Intensive Care Med 2015; 41 (08) 1411-1423
• 22 Morimoto T, Gandhi TK, Seger AC, Hsieh TC, Bates DW. Adverse drug events and medication errors: detection and classification methods. Qual Saf Health Care 2004; 13 (04) 306-314
• 23 Wong A, Amato MG, Seger DL. , et al. Evaluation of medication-related clinical decision support alert overrides in the intensive care unit. J Crit Care 2017; 39: 156-161
• 24 Wong A, Amato MG, Seger DL. , et al. Prospective evaluation of medication-related clinical decision support over-rides in the intensive care unit. [published online ahead-of-print February 9, 2018] BMJ Qual Saf 2018;
• 25 Gandhi TK, Weingart SN, Seger AC. , et al. Outpatient prescribing errors and the impact of computerized prescribing. J Gen Intern Med 2005; 20 (09) 837-841
• 26 Bobb A, Gleason K, Husch M, Feinglass J, Yarnold PR, Noskin GA. The epidemiology of prescribing errors: the potential impact of computerized prescriber order entry. Arch Intern Med 2004; 164 (07) 785-792
• 27 Bates DW, Kuperman GJ, Wang S. , et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530
• 28 Tolley CL, Slight SP, Husband AK, Watson N, Bates DW. Improving medication-related clinical decision support. Am J Health Syst Pharm 2018; 75 (04) 239-246
• 29 Carver CS, Scheier MF. The self-attention-induced feedback loop and social facilitation. J Exp Soc Psychol 1981; 17: 545-568
• 30 Carver CS, Scheier MF. Control theory: a useful conceptual framework for personality-social, clinical, and health psychology. Psychol Bull 1982; 92 (01) 111-135
• 31 Lesar TS, Briceland LL, Delcoure K, Parmalee JC, Masta-Gornic V, Pohl H. Medication prescribing errors in a teaching hospital. JAMA 1990; 263 (17) 2329-2334