Keywords adverse drug events - natural language processing - clinical decision support systems
Background and Significance
Background and Significance
Adverse drug reactions (ADRs) comprise a serious health care problem, causing substantial morbidity and mortality.[1 ]
[2 ]
[3 ] They generate preventable emergency department visits and hospital admissions, prolong hospital stays, and increase health care costs.[1 ]
[2 ]
[3 ]
[4 ]
[5 ]
[6 ]
[7 ] Providers often fail to recognize ADRs quickly enough or at all.[1 ]
[2 ] As new drugs become available and patients take more medications, providers will face increasing challenges in recognizing their patients' ADRs. This article describes the development and evaluation of the Adverse Drug Effect Recognizer (ADER). Authors hypothesized that ADER could help physicians recognize and address patients' ongoing preadmission ADRs.
Forty years ago, the World Health Organization[8 ] (WHO) defined ADR as “a response to a drug that is noxious and unintended and occurs at doses normally used in man for the prophylaxis, diagnosis or therapy of disease, or for modification of physiological function.” Edwards and Aronson[9 ] later defined an ADR as “an appreciably harmful or unpleasant reaction, … related to the use of a medicinal product, which predicts hazard from future administration and warrants prevention or specific treatment, or alteration of the dosage regimen, or withdrawal of the product.” They explained that adverse drug effect and ADR refer to the same concept, but while ADR is from the point of view of the patient, adverse drug effect is from the point of view of the drug.[9 ] The ADER system identifies electronic health record (EHR)-based patient findings that match known adverse effects of patients' preadmission medications. Only the patient's physician can determine whether matched findings actually represent true ADRs, since a patient's underlying disorder might alternatively have caused the finding. For clarity, the authors use the term adverse effect for known medication side effects. Authors use ADR for clinician-confirmed true ADRs. Correspondingly, authors refer to findings mentioned in the EHR that match known adverse effects as potential ADRs .
A 2008 systematic review of 25 studies found that, on average, 5% of hospital admissions were associated with ADRs.[5 ] A 2014 retrospective analysis of 7 million U.S. hospital discharge abstracts similarly found a combined prevalence of 5.6% for preadmission and inpatient ADRs.[10 ] The Centers for Disease Control and Prevention (CDC) noted that 82% of American adults take at least one medication, and 29% take five or more.[2 ] A 2014 study found those taking three or more medications were at higher risk for ADRs, with an increased risk for those taking more medications.[11 ] Additionally, the CDC estimated that ADRs result in more than one million emergency department visits and 280,00 hospitalizations in the United States annually.[1 ]
[2 ] Thus, ADRs generate substantial health care costs.[12 ]
[13 ]
[14 ]
[15 ]
[16 ] In 2006, the Institute of Medicine estimated that the U.S. spent $3.5 billion on extra medical costs due to ADRs, and that 40% of the costs of outpatient ADRs were preventable.[2 ]
Over the past three decades, investigators have developed automated approaches to ADR detection and prevention. Early systems identified ADRs that clinicians already had recognized (and presumably addressed). Those systems surveyed inpatient records for sudden discontinuations of medications, administrations of known antidotes, and billing codes indicating ADRs.[17 ]
[18 ] In the 1990s, University of Utah researchers implemented such a system; it detected significantly more verified ADRs than traditional reporting methods.[19 ]
[20 ]
[21 ] Bates et al later showed that automated ADR detection using diagnosis codes, allergy rules, and text analysis were effective, although with lower predictive values than intensive retrospective manual chart reviews.[22 ]
[23 ]
[24 ] Nevertheless, they also found that combining discharge summaries, laboratory results, and medication records could identify valuable ADR-related information.[25 ]
[26 ]
[27 ] Their text-searching tools revealed more potential ADRs than simply reviewing laboratory results or diagnostic codes.[23 ]
Since 2000, improved natural language processing (NLP) techniques have enabled researchers to better identify ADRs from clinical documents. For example, researchers from Columbia University applied NLP to discharge summaries and the biomedical literature to retrospectively detect potentially unrecognized ADRs and other disease–drug associations.[28 ]
[29 ]
[30 ]
[31 ]
[32 ] Researchers at Stanford University also utilized NLP to extract ADR information for pharmacovigilance studies.[33 ]
[34 ]
[35 ]
[36 ]
[37 ] The latter studies demonstrated the need to apply additional drug knowledge to distinguish ADRs from confounders such as indications for drug therapy.
Past work utilizing NLP in combination with data on known adverse effects forms the conceptual basis for the current ADER study. Previous studies[38 ]
[39 ] have validated NLP and text-mining methods for retrospectively identifying adverse effects from EHR notes. On admission, ADER automatically extracts inpatients' medications, symptoms, and previous diagnoses using NLP of provider-generated admission history and physical examination (H&P) notes. The system then cross-references these data against a set of known adverse effects and generates patient-specific alerts in the EHR. This article describes an intervention study to determine whether ADER could improve providers' recognition of ADRs. The current study is one of the first to evaluate a system that uses NLP to warn clinicians of potential ongoing ADRs at the time of hospital admission.
Objective
The goal of this project was development of the ADER system and preliminary evaluation of whether it could improve inpatient providers' recognition of ongoing ADRs due to outpatient medications. Authors hypothesized that physicians receiving ADER alerts would hold or discontinue suspected ADR-causing medications at a greater rate than a retrospective control group of similar physicians who did not receive alerts. The study assessed ADER's effect on ADR recognition by comparing admission medications to inpatient and discharge medication orders for patients in both the intervention and control groups.
Methods
The ADER system utilizes several previously validated and publicly available NLP tools. SecTag[40 ] identifies section headers in H&P notes; the KnowledgeMap Concept Identifier (KMCI)[41 ]
[42 ] extracts clinical concepts from text in a manner similar to MetaMap,[43 ] and MedEx[44 ] extracts medication names and dosage information from clinical texts. MedEx was originally trained on data similar to that used for the ADER evaluation.[44 ] A previous study confirmed that H&P notes at the authors' institution accurately capture patients' preadmission medications.[45 ] In the current study, KMCI and ADER used the 2013ab version of the Unified Medical Language System (UMLS); ADER and MedEx utilized the 3 February 2014 release of RxNorm.[46 ]
[47 ]
ADER Design
Upon each patient's hospital admission, clinicians perform a thorough evaluation recorded in the H&P notes. During the current study, immediately upon generation of a new H&P in the hospital EHR, ADER copied the note to its database for processing. First, ADER ran SecTag to parse sections of the note, such as Chief Complaint , History of Present Illness , Past Medical History , Medications , and Family Medical History . Next, ADER used KMCI to identify the patient's symptoms, findings, and previous diagnoses (diseases) mentioned in each section and mapped them to the Systematized Nomenclature of Medicine – Clinical Terms (SNOMED-CT) clinical vocabulary via UMLS.[48 ] Using an implementation of the NegEx algorithm,[49 ] KMCI identified whether any identified concepts were characterized as absent (e.g., “no fever,” “patient denied cough,” etc.). The ADER system ignores such negated items and also terms mentioned in sections unlikely to represent the patient's own findings (e.g., Family History as identified by SecTag). Additionally, ADER used regular expressions to extract numerical values from the H&P Vital Signs section, mapping abnormal findings to SNOMED-CT (e.g., hypotension for systolic blood pressure < 90). Similarly, ADER mapped abnormal admission of EHR laboratory results to SNOMED-CT concepts (e.g., hypokalemia for serum potassium < 3.2 mEq/L). Next, ADER utilized MedEx to identify the patient's current medications from the H&P's Medications section. It automatically mapped drugs to their generic ingredients and represents them using RxNorm.[47 ]
Finally, ADER cross-referenced the patient's recorded medications and clinical findings against a database of known adverse effects. Thus, ADER identified potential ongoing ADRs due to the patient's preadmission (outpatient) medications. The ADER adverse effect database derives from the authors' previous work on the Drug Evidence Base (DEB2), a knowledge base containing drug indications and adverse effects extracted from five public domain sources using automated methods.[50 ] See details in the “ADER Evaluation” section below.
A patient's underlying medical conditions can also potentially explain findings that ADER labels as potential ADRs. For example, a patient taking Lisinopril might have a cough due to chronic obstructive pulmonary disease, and not due to an ADR. Prior to the current study, the authors calculated the frequency of mentions of each potential ADR finding in a corpus of 350,000 historical H&P notes. Authors then discerned how often such mentions occurred in patients taking potentially ADR-causing medications, as well as how often such mentions were associated with patients' various discharge diagnoses when no ADR-causing medication was recorded. Based on those calculated relative risk scores, ADER determined whether a patient's current diagnoses plausibly might have caused a finding that ADER identified as a potential ADR. The system warned clinicians of such confounding diseases in its ADR alerts. [Fig. 1 ] provides an overview of ADER workflow.
Fig. 1 Complete Adverse Drug Effect Recognizer (ADER) workflow diagram. HPI, history of present illness; PE, physical exam; ROS, review of systems.
Within a few minutes of a clinician creating an H&P note, ADER identified any potential ADRs within the note and generated patient-specific alerts. Each alert displayed the potential ADRs, suspected causative medications, specific evidence extracted from the chart, and any possible confounding diseases. [Fig. 2 ] shows sample ADER alerts. Physicians could review the alerts in multiple places in Vanderbilt's EHR, “StarPanel”:[51 ] the EHR Team Summary (used by all care providers); in StarPanel EHR progress note template forms (used to create clinicians' daily progress notes); and, in frequently used, patient-specific, paper-based reports (used by physicians for daily rounds). As part of the current study, alerts included a brief embedded questionnaire to collect feedback from intervention providers.
Fig. 2 Several Adverse Drug Effect Recognizer (ADER) alerts from the study, including potential false positive alerts (renal insufficiency, cough, and edema are potential ADRs, but could also be caused by the patient's kidney failure, chronic obstructive pulmonary disease [COPD], and heart failure).
To prevent alert fatigue,[52 ] and also at the request of physicians contributing to study design, ADER used noninterruptive alert delivery mechanisms (i.e., its passive displays did not break EHR workflows). The system automatically removed alerts from the EHR when: (1) a provider responded to the alert questionnaire; (2) the patient was discharged; or (3) alert-related questionnaire went unanswered for 5 days.
ADER Evaluation
The ADER intervention study tested the hypothesis that ADER could help providers to better recognize and respond to inpatients' preadmission ongoing ADRs. The study focused on one common class of medications, antihypertensives, which have a myriad of known side effects. The complete set of antihypertensive adverse effects detectable by ADER appears in the Supplementary Material ([Supplementary Tables S1 ]
[S2 ]
[S3 ], available in the online version). A team of clinical pharmacists reviewed the antihypertensive adverse effects from the previously mentioned DEB2 database, making any necessary clinically relevant revisions prior to initiation of the study. After the target adverse effects had been determined, the authors utilized the aforementioned corpus of 350,000 previous-to-current-study H&Ps to refine KMCI's recognition of those specific effects. This process captured common misspellings, abbreviations, and lexical variants, which were used to improve KMCI recognition of them. To determine the accuracy of concepts recognized by ADER's NLP during the study, the authors analyzed a sample of 100 H&P notes where alerts had been issued. This manual audit attempted to verify that all ADER-identified clinical findings and medications mentioned in alerts were actually present in patient's records.
The Vanderbilt University Institutional Review Board (IRB) approved the study (IRB #141341). Indirectly, the study involved participating house officers' attending physicians and adult inpatients; they did not require informed consent because the intervention simply informed house officers of potential ADRs. The system did not directly change patients' medications. All treatment and medication decisions remained entirely with the care team.
The ADER intervention study began on August 1 and ended on October 31, 2015. The intervention group included consented interns and residents on the General Internal Medicine wards of the Vanderbilt University Hospital. The study length was based on pragmatic convenience—authors were uncertain prior to the study whether the number of alerts that ADER generated would cause alert fatigue. Therefore, consented participating residents and interns, as well as the Director of the Internal Medicine Residency program, agreed to a 3-month trial period as the maximum tolerable duration (in the event that warnings became onerous). The retrospective control group consisted of Internal Medicine interns and residents assigned to the same wards as intervention physicians during a 3-month period just prior to the activation of the system (April 1 through June 30, 2015). To enable comparison of intervention physicians' behaviors to that of controls, the authors ran ADER on the retrospective control group's previous H&P notes, identifying instances when alerts would have been issued had the system been active at the time.
For both intervention and control groups, the study analyzed all clinicians' prescribing behaviors using NLP-based analysis of admission notes, inpatient medication orders, and discharge summaries. This process determined whether preadmission medications from the H&P were continued, held or discontinued during the hospital stay and at discharge. The study compared medication hold/discontinue rates of all intervention clinicians who received alerts and all control physicians who would have received alerts had the system been active. The study also determined separately whether the subgroup of intervention providers who completed the alert-embedded survey (and therefore definitely had seen the alerts) were more likely to change suspected ADR-causing medications compared with all nonalerted control group providers. When evaluating discharge medications, the study only analyzed notes for those patients both admitted and discharged by study subjects (i.e., Internal Medicine house staff). Patients transferred to the care of physicians on another service (e.g., Surgery) were not included in analysis of discharge medications.
When comparing preadmission medications to inpatient medication records , the authors evaluated the difference between intervention and control groups for suspected ADR-causing medications (1) ordered ever during the inpatient stay, (2) ordered in the first 24 hours of the stay, (3) ordered, then stopped or held, as well as (4) suspected ADR-causing medications stopped or held after intervention providers responded to the alert survey versus the overall hold rate in control group. When comparing preadmission medications to discharge summaries , the authors evaluated the difference between intervention and control groups for suspected ADR-causing medications (5) held at discharge and (6) held at discharge for intervention providers who responded to the alert survey. The primary outcome was (1), the difference between groups for suspected ADR-causing medications ordered ever during the inpatient stay.
Whenever intervention subjects' survey responses indicated that they would make or had made changes to suspected ADR-causing medications, the study identified (by analyzing actual orders issued) whether the provider actually followed-through with the medication order changes. An original long version of the survey questionnaire had a low response rate (12%); subjects reported that it was cumbersome and had too many questions. Therefore, a shorter survey ([Fig. 2 ]) replaced the longer version approximately 3 weeks into the study. Only survey responses obtained during the 9 weeks using the shorter-form survey appear in the results. The authors reviewed open-ended survey comments submitted by intervention subjects and classified them based upon similar themes. Additionally, the authors met with study subjects, including the Chief Resident and Director of the Internal Medicine Residency Program, to discuss their experiences and suggestions at the completion of the intervention.
Statistical Analysis
The study characterized categorical variables using frequencies and proportions and used medians and interquartile range (IQR) to describe continuous variables. To determine whether ADER alerts affected providers' behaviors, authors analyzed medication order changes during the inpatient stay and at discharge. The analysis compared the rates of medication changes between the control and intervention groups using chi-square tests with a significance level of 0.05. When performing secondary analyses on data, the authors used Bonferroni correction to adjust the significance level for all comparisons to account for multiple testing. All statistical analyses used R version 2.15.2.
Results
[Table 1 ] illustrates ADER study results for both the intervention and retrospective control groups. The 3-month preintervention control group included 2,666 H&P notes written by 138 interns and residents. During the 3-month intervention period, the ADER scanned 2,312 H&P notes written by 137 interns and residents. No significant differences occurred between the intervention and control groups for any metrics shown in [Table 1 ] (p > 0.05). The ADER system detected potential ADRs in 30 to 32% of both groups' notes, with 78 to 79% triggered by H&P text findings and 21 to 22% by laboratory results. The median time from when an intervention group physician saved an admission note to the EHR to when ADER posted an alert was 8 minutes (IQR = 6–9 minutes). The median time between ADER issuing an alert and an intervention subject responding to the embedded survey questions was 24 hours (IQR = 17–37 hours).
Table 1
ADER study results for intervention and control groups
H&Ps
Unique patients
Notes with potential ADRs
Average number of antihypertensive medications per note
N
N
%
N
%
Notes without potential ADRs
Notes with potential ADRs
Intervention
2,312
2,049
89
701
30
0.73
2.4
Control
2,666
2,352
88
867
32
0.76
2.4
Potential ADRs per alert
Adverse effect source
Number of potential ADRs found per alert
H&P text
Labs
Min.
25%
Median
75%
Max
Intervention
2.3
79
21
1
1
2
3
15
Control
2.4
78
22
1
1
2
3
23
Admissions with discharge note
Unique patients with discharge note
Average length of stay
Number of providers
N
%
N
%
Total
With discharge
Intervention
1,835
79
1,646
90
115 h
137
137
Control
2,160
81
1,916
89
112 h
138
128
Abbreviations: ADER, Adverse Drug Effect Recognizer; ADR, adverse drug reaction; H&P, history and physical examination.
ADER NLP Accuracy
Manual expert review of 100 intervention group H&P notes for which ADER had issued potential ADR alerts revealed that 98% of the 157 medications that ADER identified as current therapies were correct. In three instances, ADER incorrectly identified drugs that the H&Ps described as previously discontinued. Of 136 total adverse effects (findings or diseases) that ADER identified from H&P notes, 116 (86%) were confirmed as “present” in the chart. Of the 20 incorrectly identified adverse effects, 15 were mentioned as being absent but were not recognized as “negated” by KMCI due to use of nonstandard wording. The remaining misidentified concepts were mentioned only as possible points of differential diagnosis or past inactive diagnoses. Of the 30 clinical findings the ADER recognized from laboratory results, 29 (97%) were accurate. The only incorrectly classified numerical laboratory result was “hyperkalemia” that had been noted in comments to be due to a hemolyzed sample.
Comparing Admission Medications to Inpatient Medication Orders
The intervention group ordered significantly fewer suspected ADR-causing preadmission medications at any point during the inpatient stay compared with the control group (47% vs. 58%, p < 0.001). Similarly, during the first 24 hours after admission, intervention physicians ordered only 28% of 1,140 suspected ADR-causing preadmission medications flagged by ADER; the control group had ordered 39% of 1,468 suspected ADR-causing preadmission medications during the same interval (28% vs. 39%, p < 0.001). See [Table 2 ].
Table 2
Preadmission medications compared with ordered inpatient medications, testing the null hypothesis that the proportion for the intervention group was equal to the proportion for the control group
Suspected ADR-causing medications
Ordered ever during inpatient stay
Ordered in first 24 hours
Ordered, then stopped or held
N
Proportion
N
Proportion
N
Proportion
Intervention
1,140
532
0.467
315
0.276
91
0.171
Control
1,468
846
0.576
570
0.39
125
0.148
Difference
(95% CI)
p -value
–0.109
(–0.149, –0.070)
p < 0.001
–0.114
(–0.149, –0.075)
p < 0.001
0.023
(–0.018, 0.065)
p = 0.28
Abbreviations: ADR, adverse drug reaction; CI, confidence interval.
Among suspected ADR-causing medications ordered during the inpatient stay, 84% (711/846) were ordered within 24 hours of admission in the control group; in the intervention group, 77% (410/532) were ordered within 24 hours of admission (p = 0.002). Suspected ADR-causing medications were stopped before patient discharge at rates of 14 and 16% for the control and intervention groups, respectively (p = 0.40). Among medications started after 24 hours into the inpatient stay, subsequent hold rates were 19 and 20% in the control and intervention groups, respectively (p = 1.00).
Among suspected ADR-causing preadmission medications ordered at any point in the inpatient stay, the rate at which they were subsequently held or discontinued did not vary significantly between groups (see [Table 2 ], p = 0.28). However, secondary analysis, as shown in [Table 3 ], revealed that the subset of intervention providers who responded to alert-embedded surveys—and therefore who definitely viewed the alerts—subsequently held or discontinued 25% (41/167) of the then-active suspected ADR-causing preadmission medications. This rate is significantly different than the aforementioned control group rate (25% vs. 15%, p = 0.003). In the intervention group, the median time between responding to the survey and subsequent medication changes was 25 hours (IQR = 13–66).
Table 3
Preadmission medications ordering during inpatient stay later stopped or held, testing the null hypothesis that the proportion for the intervention group after survey response was equal to the overall proportion for the control group
Suspected ADR-causing meds
N
N
Proportion
Intervention
Active order at time of ADER alert survey response
167
Stopped or held after survey response
41
0.246
Control
Ordered ever during the inpatient stay
846
Ordered ever, then stopped or held
125
0.148
Difference
(95% CI)
p -value
0.098
(0.025, 0.171)
p = 0.003
Abbreviations: ADER, Adverse Drug Effect Recognizer; ADR, adverse drug reaction; CI, confidence interval.
Comparing Admission Medications to Discharge Medications
Approximately 20% of intervention and control group discharge summaries were not generated by study subjects and thus were excluded from review. [Table 4 ] shows hold/discontinuation rates of suspected ADR-causing preadmission medications at discharge for the remaining 80% of discharge summaries.
Table 4
Preadmission medications compared with discharge medications testing the null hypothesis that the proportion for the intervention group was equal to the proportion for the control group
Suspected ADR-causing preadmission medications: full intervention group versus full control group
Suspected ADR-causing preadmission medications; intervention group survey respondents versus full control group
Total
Medication missing from discharge note
Held or discontinued at discharge (explicitly)
Total
Held or discontinued at discharge (explicitly)
N
N
Proportion
N
Proportion
N
N
Proportion
Intervention
927
104
0.112
184
0.198
258
97
0.376
Control
1,225
75
0.061
287
0.234
1,225
287
0.234
Difference
(95% CI)
p -value
0.051 (0.025, 0.076)
p < 0.001
–0.036 (–0.072, 0.0001)
p = 0.053
0.142
(0.076, 0.208)
p < 0.001
Abbreviations: ADR, adverse drug reaction; CI, confidence interval.
When discharge notes did not mention previously alerted preadmission medications, the study could not definitively determine whether the medications had been held or stopped at discharge (104/927 = 11% from intervention, 75/1,225 = 6.1% from control, p < 0.001). Actual specific mentions in discharge notes of holds/discontinuations for alerted medications did not differ significantly: 20% (184/927) in the intervention group and 23% (287/1225) in the control group (p = 0.053). However, intervention providers who responded to the alert-embedded survey held or discontinued 38% (97/258) of suspected ADR-causing medications at discharge, significantly different (p < 0.001) than the overall 23% rate for the control group.
Survey Questionnaire Results and Provider Comments
Intervention subjects responded to 53% (298/560) of the short-version alert-embedded questionnaires (see [Table 5 ]). For survey respondents who stated that they would change the patient's therapy due to identified ADRs, subsequent medication holds or dosage decreases occurred more than 90% of the time. For the 38 intervention subjects who responded that they were uncertain whether they would make a change, analysis identified 24 instances (63%) where one or more suspected ADR-causing medications was subsequently changed. Nearly three-quarters of survey responses indicated alerts either somewhat or fully merited consideration for their patients. More than half said alerts were helpful or possibly helpful in managing the patient.
Table 5
Responses to ADER alert survey questionnaires (n = 298)
Question
Response
N
(%)
N
(%)
N
(%)
Did any of these alerts merit at least passing consideration for this patient?
Yes
144
(48)
Somewhat
74
(24)
No
80
(26)
Were any of these alerts helpful in managing this patient?
Yes
87
(29)
Possibly
80
(26)
No
131
(43)
Will you (or did you already) change patient's therapy due to these possible ADRs?
Yes
91
(30)
Uncertain
38
(12)
No
169
(56)
Abbreviations: ADER, Adverse Drug Effect Recognizer; ADR, adverse drug reaction.
Approximately 30% of the 52 optional free-text survey comments indicated that ADER was correct in identifying an ADR. Conversely, 30% of the survey comments stated that claimed adverse effects were more likely due the patient's underlying medical conditions (usually those recognized by ADER as possible confounders). For example, when ADER identified renal insufficiency or acute kidney injury as an ADR, providers often indicated that the patient's existing end-stage renal disease was the more likely cause of impaired renal function.
In ∼20% of survey comments, providers indicated that the medications were held, but not due to the ADER-recognized ADRs. In at least two cases, however, the H&P Plan section indicated that a provider had in fact held the medications for the reasons identified by ADER. In 10% of the comments, providers indicated that the patients were not suffering from the suspected ADR. In one of those cases, ADER had recognized “gout” as a possible ADR. The patient had a previous diagnosis of gout, but no acute gout exacerbation was present on admission. In another case, ADER “incorrectly” identified that the patient had a cough because of the phrase, “Hurts with movement, cough, sneezing, position changes.” Several survey respondents suggested that they had already recognized the ADR suggested by ADER.
Study subjects also provided feedback to the authors directly. Participating providers generally viewed ADER favorably, but suggested that ADER might be more useful for those medications that internists prescribe less frequently than antihypertensives, such as psychiatric medications. They also expressed that potential confounders included in ADER alerts were helpful in assessing the veracity of suspected ADRs and in considering whether medications were possibly exacerbating the recognized findings.
Medication Holds Analyzed by ADR
Among the 940 intervention group alerts, 41 were distinct adverse effects (patient findings/disorders). The most frequent alert-related conditions (occurring more than 50 times) were: renal insufficiency, light headedness, edema, dizziness, hypotension, and syncope. Suspected causative medications were held at discharge at rates of 35% for hypotension, 28% for renal insufficiency, 42% for hyperkalemia, 36% for hypokalemia, and 50% for hyponatremia. The Supplementary Material ([Supplementary Tables S1 ]
[S2 ]
[S3 ], available in the online version) provides hold/discontinue rates for all medications with detected ADRs. The online Supplementary Material also reports comparisons of hold rates between suspected ADR-causing medications and those not suspected of causing ADRs.
Discussion
The current study illustrates the potential for automated alerting systems like ADER to help providers identify unrecognized ADRs. Physicians receiving ADER alerts held or discontinued more suspected ADR-causing medications at multiple points in the patient care workflow compared with physicians who did not receive alerts. In survey responses, physicians stated that ADER alerts more than half the time helped with patient management. Analysis of ADER's accuracy revealed that the majority EHR-extracted findings and medications were correct, but also revealed opportunities for improvements in NLP, particularly in regards to negation detection.
Intervention providers receiving ADER alerts ordered significantly fewer suspected ADR-causing preadmission medications in the first 24 hours after admission and at any time during the inpatient stay. As a whole, intervention providers receiving alerts were no more likely to discontinue suspected ADR-causing preadmission medications at discharge than the control group (although they had ordered fewer such medications overall at admission). Nevertheless, providers who responded to the alert-embedded survey were more likely to hold or discontinue ADR-causing medications during the patient's hospitalization and at discharge. Two explanations are possible—either providers who believed that a potential ADR alert was valid were more likely to respond to the survey, or providers who believed ADER alerts were true more often addressed potential ADR-causing medications at discharge. Further study could determine whether ADER alerts would be more effective if they interrupted workflows, since the current study used noninterruptive alerts. Why intervention physicians in some instances ordered previously alerted ADR-causing medications later during an admission is also a topic for further research. A possible explanation is that when ADR-related findings resolved, re-starting the medication was appropriate (e.g., hypotension in patients with baseline hypertension).
From a technical standpoint, the ADER approach is generalizable to many inpatient facilities. The system only requires that a hospital have an EHR system that can electronically export (under appropriate security) admission H&P notes and laboratory values as they are generated to a generic desktop-class computer running the ADER software. Another key requirement is the hospital EHR's ability to incorporate ADER-generated alerts into the EHR workflow in locations likely to be viewed by clinicians.
Comparison with Prior Research
Use of NLP to identify potential ongoing ADRs from outpatient medications at the time of admission is novel. A substantial body of work exists regarding other methods of ADR detection and associated determinations of physicians' responses. In 2017, Dexheimer et al reported that providers in a pediatric hospital, after receiving inpatient dosing, allergy, and drug–drug interaction alerts cancelled or modified medication orders an average of 12.6, 7.0, and 16.6%, respectively.[53 ] A 2003 study of five adult primary care practices found that physicians accepted only 8.8% of drug allergy and 10.6% of high-severity drug–drug interaction alerts.[54 ] Further analysis revealed physicians were more likely to override alerts for renewals than new prescriptions and that house officers were less likely to prescribe alerted medications than other physicians. A different study suggested that house staff were more likely to override ADR alerts than staff physicians.[55 ] In a classic 1976 study, McDonald evaluated provider responses to electronic reminders, including suggestions to observe a physical finding or ask about a specific symptom, order a diagnostic study or change or start a therapeutic regimen.[56 ] The study found physicians responded to 51% of events when given electronic reminders compared with 22% of events when they did not receive reminders. Of note, few such reminders involved medication changes due to potential ADRs. The study indicated that physicians comply with a higher proportion of electronic recommendations when protocols are more precise.
The current study's physician alert response rates exceed the aforementioned previous studies regarding other types of ADR alerts. A possible explanation is that the current study named in its alerts the suspected ADR symptoms that were present in the patient at admission. The ADER methodology is significantly different than that of the above-mentioned efforts, as well as the adverse effect detection studies discussed in the background. The previous efforts utilized sudden discontinuations of medications, administrations of known antidotes, and ADR-related billing codes, all of which indicate that providers had already recognized the ADRs. They also warned about dosing errors, allergies, and possible drug–drug interactions, which often do not take into account the actual symptoms being experienced by the patient. The ADER system is novel in that it utilizes NLP of provider-generated admission notes to warn clinicians about potential ongoing ADRs, as evidenced by documented symptoms and findings, due to outpatient medications. It has the potential to detect unrecognized ADRs and provide a safeguard to ensure all providers are aware of the ADRs their patients may be experiencing.
Limitations and Future Work
Several limitations pertain to the study. The intervention only included Internal Medicine interns and residents as alert recipients. That survey responses followed alerts by ∼24 hours suggests that house officers may have waited to discuss alerts with the ward team and attending physicians before acting. Additionally, house staff often rotate “off service” so that the physicians writing discharge orders may not have been the ones who received the initial ADER alerts at the time of admission, even though the discharging physician qualified as study participants by membership in the Internal Medicine house staff. A wider range of subjects (e.g., academic attending physicians, community-based physicians, nurse practitioners, physician assistants) should be evaluated in teaching and nonteaching settings. The study limited the ADER intervention to antihypertensive medications so that potential alert fatigue would not confound initial evaluation of ADER's utility—as might occur if all categories of medications generated alerts. The study targeted the commonly prescribed antihypertensive class because they cause a wide variety of ADRs. Admittedly, providers who regularly prescribe these medications are more likely to be aware of their potential ADRs, and study participants suggested that ADER might be more useful for medications they prescribe less frequently. In the future, ADER should be expanded, in consultation with clinical experts, to cover additional drug classes and their related ADRs, with particular focus on uncommon or severe ADRs.
The study analyzed 3 months of intervention data compared with a similar-duration retrospective control group. While analysis showed the control group to be comparable to the intervention group in nearly all respects, the groups contained different participants and the admissions occurred at somewhat different times in the academic year (controls were soon-to-be promoted house officers; intervention subjects had arrived in their current positions 1 month prior to study initiation). A randomized controlled trial involving multiple institutions and all drug classes could provide a more definitive result than the current study. Before undertaking such a study, the potentially correctable flaws in ADER's NLP methods (identified in this study) should be addressed.
To evaluate the potential of this methodology, this study focused only on the accuracy of the ADER alerts that were presented. Future studies with additional drug classes should consider both precision and recall to determine which potential ADRs ADER misses (i.e., fails to generate alerts when appropriate to warn clinicians). To reduce alert fatigue, more advanced confounder detection should be studied with the goal of filtering out false-positive alerts when potential ADRs have a significantly more likely clinical cause. Future studies will also include more in-depth qualitative analysis of providers' impressions and resultant behavior.
Conclusion
Intervention subjects reduced ordering of suspected ADR-causing medications at multiple points during the admission compared with the control group. Systems like ADER have the potential to improve both recognition of adverse effects and discontinuation of medications causing preadmission ADRs. Future ADER-like systems must supply relevant information to care providers at the right place and time to improve patient safety, potentially decrease cost, and improve the quality of care.
Clinical Relevance Statement
Clinical Relevance Statement
This study shows that using NLP extraction of patient preadmission medications and findings to generate real-time ADR alerts can affect provider ordering behavior. This approach could be generalized to other facilities that use electronic health records. Clinical decision support systems like ADER could help to improve the safety of care delivery in the future.
Multiple Choice Questions
Multiple Choice Questions
Approximately what percentage of ADER-identified adverse drug reactions were found in the text of the admission note, as compared with solely derived from laboratory test results?
How often did intervention clinicians receiving an alert order the potentially offending medication during the hospital stay, and how often did control physicians do so?
