Evaluation of a Novel System to Enhance Clinicians' Recognition of Preadmission Adverse Drug Reactions

Abstract

Background Often unrecognized by providers, adverse drug reactions (ADRs) diminish patients' quality of life, cause preventable admissions and emergency department visits, and increase health care costs.

Objective This article evaluates whether an automated system, the Adverse Drug Effect Recognizer (ADER), could assist clinicians in detecting and addressing inpatients' ongoing preadmission ADRs.

Methods ADER uses natural language processing to extract patients' medications, findings, and past diagnoses from admission notes. It compares excerpted information to a database of known medication adverse effects and promptly warns clinicians about potential ongoing ADRs and potential confounders via alerts placed in patients' electronic health records (EHRs). A 3-month intervention trial evaluated ADER's impact on antihypertensive medication ordering behaviors. At the time of patient admission, ADER warned providers on the Internal Medicine wards of Vanderbilt University Hospital about potential ongoing preadmission antihypertensive medication ADRs. A retrospective control group, comprised similar physicians from a period prior to the intervention, received no alerts. The evaluation compared ordering behaviors for each group to determine if preadmission medications changed during hospitalization or at discharge. The study also analyzed intervention group participants' survey responses and user comments.

Results ADER identified potential preadmission ADRs for 30% of both groups. Compared with controls, intervention providers more often withheld or discontinued suspected ADR-causing medications during the inpatient stay (p < 0.001). Intervention providers who responded to alert-related surveys held or discontinued suspected ADR-causing medications more often at discharge (p < 0.001).

Conclusion Results indicate that ADER helped physicians recognize ADRs and reduced ordering of suspected ADR-causing medications. In hospitals using EHRs, ADER-like systems could improve clinicians' recognition and elimination of ongoing ADRs.

Keywords

adverse drug events - natural language processing - clinical decision support systems

Background and Significance

Adverse drug reactions (ADRs) comprise a serious health care problem, causing substantial morbidity and mortality.[1] [2] [3] They generate preventable emergency department visits and hospital admissions, prolong hospital stays, and increase health care costs.[1] [2] [3] [4] [5] [6] [7] Providers often fail to recognize ADRs quickly enough or at all.[1] [2] As new drugs become available and patients take more medications, providers will face increasing challenges in recognizing their patients' ADRs. This article describes the development and evaluation of the Adverse Drug Effect Recognizer (ADER). Authors hypothesized that ADER could help physicians recognize and address patients' ongoing preadmission ADRs.

Forty years ago, the World Health Organization[8] (WHO) defined ADR as “a response to a drug that is noxious and unintended and occurs at doses normally used in man for the prophylaxis, diagnosis or therapy of disease, or for modification of physiological function.” Edwards and Aronson[9] later defined an ADR as “an appreciably harmful or unpleasant reaction, … related to the use of a medicinal product, which predicts hazard from future administration and warrants prevention or specific treatment, or alteration of the dosage regimen, or withdrawal of the product.” They explained that adverse drug effect and ADR refer to the same concept, but while ADR is from the point of view of the patient, adverse drug effect is from the point of view of the drug.[9] The ADER system identifies electronic health record (EHR)-based patient findings that match known adverse effects of patients' preadmission medications. Only the patient's physician can determine whether matched findings actually represent true ADRs, since a patient's underlying disorder might alternatively have caused the finding. For clarity, the authors use the term adverse effect for known medication side effects. Authors use ADR for clinician-confirmed true ADRs. Correspondingly, authors refer to findings mentioned in the EHR that match known adverse effects as potential ADRs.

A 2008 systematic review of 25 studies found that, on average, 5% of hospital admissions were associated with ADRs.[5] A 2014 retrospective analysis of 7 million U.S. hospital discharge abstracts similarly found a combined prevalence of 5.6% for preadmission and inpatient ADRs.[10] The Centers for Disease Control and Prevention (CDC) noted that 82% of American adults take at least one medication, and 29% take five or more.[2] A 2014 study found those taking three or more medications were at higher risk for ADRs, with an increased risk for those taking more medications.[11] Additionally, the CDC estimated that ADRs result in more than one million emergency department visits and 280,00 hospitalizations in the United States annually.[1] [2] Thus, ADRs generate substantial health care costs.[12] [13] [14] [15] [16] In 2006, the Institute of Medicine estimated that the U.S. spent $3.5 billion on extra medical costs due to ADRs, and that 40% of the costs of outpatient ADRs were preventable.[2]

Over the past three decades, investigators have developed automated approaches to ADR detection and prevention. Early systems identified ADRs that clinicians already had recognized (and presumably addressed). Those systems surveyed inpatient records for sudden discontinuations of medications, administrations of known antidotes, and billing codes indicating ADRs.[17] [18] In the 1990s, University of Utah researchers implemented such a system; it detected significantly more verified ADRs than traditional reporting methods.[19] [20] [21] Bates et al later showed that automated ADR detection using diagnosis codes, allergy rules, and text analysis were effective, although with lower predictive values than intensive retrospective manual chart reviews.[22] [23] [24] Nevertheless, they also found that combining discharge summaries, laboratory results, and medication records could identify valuable ADR-related information.[25] [26] [27] Their text-searching tools revealed more potential ADRs than simply reviewing laboratory results or diagnostic codes.[23]

Since 2000, improved natural language processing (NLP) techniques have enabled researchers to better identify ADRs from clinical documents. For example, researchers from Columbia University applied NLP to discharge summaries and the biomedical literature to retrospectively detect potentially unrecognized ADRs and other disease–drug associations.[28] [29] [30] [31] [32] Researchers at Stanford University also utilized NLP to extract ADR information for pharmacovigilance studies.[33] [34] [35] [36] [37] The latter studies demonstrated the need to apply additional drug knowledge to distinguish ADRs from confounders such as indications for drug therapy.

Past work utilizing NLP in combination with data on known adverse effects forms the conceptual basis for the current ADER study. Previous studies[38] [39] have validated NLP and text-mining methods for retrospectively identifying adverse effects from EHR notes. On admission, ADER automatically extracts inpatients' medications, symptoms, and previous diagnoses using NLP of provider-generated admission history and physical examination (H&P) notes. The system then cross-references these data against a set of known adverse effects and generates patient-specific alerts in the EHR. This article describes an intervention study to determine whether ADER could improve providers' recognition of ADRs. The current study is one of the first to evaluate a system that uses NLP to warn clinicians of potential ongoing ADRs at the time of hospital admission.

Objective

The goal of this project was development of the ADER system and preliminary evaluation of whether it could improve inpatient providers' recognition of ongoing ADRs due to outpatient medications. Authors hypothesized that physicians receiving ADER alerts would hold or discontinue suspected ADR-causing medications at a greater rate than a retrospective control group of similar physicians who did not receive alerts. The study assessed ADER's effect on ADR recognition by comparing admission medications to inpatient and discharge medication orders for patients in both the intervention and control groups.

Methods

The ADER system utilizes several previously validated and publicly available NLP tools. SecTag[40] identifies section headers in H&P notes; the KnowledgeMap Concept Identifier (KMCI)[41] [42] extracts clinical concepts from text in a manner similar to MetaMap,[43] and MedEx[44] extracts medication names and dosage information from clinical texts. MedEx was originally trained on data similar to that used for the ADER evaluation.[44] A previous study confirmed that H&P notes at the authors' institution accurately capture patients' preadmission medications.[45] In the current study, KMCI and ADER used the 2013ab version of the Unified Medical Language System (UMLS); ADER and MedEx utilized the 3 February 2014 release of RxNorm.[46] [47]

ADER Design

Upon each patient's hospital admission, clinicians perform a thorough evaluation recorded in the H&P notes. During the current study, immediately upon generation of a new H&P in the hospital EHR, ADER copied the note to its database for processing. First, ADER ran SecTag to parse sections of the note, such as Chief Complaint, History of Present Illness, Past Medical History, Medications, and Family Medical History. Next, ADER used KMCI to identify the patient's symptoms, findings, and previous diagnoses (diseases) mentioned in each section and mapped them to the Systematized Nomenclature of Medicine – Clinical Terms (SNOMED-CT) clinical vocabulary via UMLS.[48] Using an implementation of the NegEx algorithm,[49] KMCI identified whether any identified concepts were characterized as absent (e.g., “no fever,” “patient denied cough,” etc.). The ADER system ignores such negated items and also terms mentioned in sections unlikely to represent the patient's own findings (e.g., Family History as identified by SecTag). Additionally, ADER used regular expressions to extract numerical values from the H&P Vital Signs section, mapping abnormal findings to SNOMED-CT (e.g., hypotension for systolic blood pressure < 90). Similarly, ADER mapped abnormal admission of EHR laboratory results to SNOMED-CT concepts (e.g., hypokalemia for serum potassium < 3.2 mEq/L). Next, ADER utilized MedEx to identify the patient's current medications from the H&P's Medications section. It automatically mapped drugs to their generic ingredients and represents them using RxNorm.[47]

Finally, ADER cross-referenced the patient's recorded medications and clinical findings against a database of known adverse effects. Thus, ADER identified potential ongoing ADRs due to the patient's preadmission (outpatient) medications. The ADER adverse effect database derives from the authors' previous work on the Drug Evidence Base (DEB2), a knowledge base containing drug indications and adverse effects extracted from five public domain sources using automated methods.[50] See details in the “ADER Evaluation” section below.

A patient's underlying medical conditions can also potentially explain findings that ADER labels as potential ADRs. For example, a patient taking Lisinopril might have a cough due to chronic obstructive pulmonary disease, and not due to an ADR. Prior to the current study, the authors calculated the frequency of mentions of each potential ADR finding in a corpus of 350,000 historical H&P notes. Authors then discerned how often such mentions occurred in patients taking potentially ADR-causing medications, as well as how often such mentions were associated with patients' various discharge diagnoses when no ADR-causing medication was recorded. Based on those calculated relative risk scores, ADER determined whether a patient's current diagnoses plausibly might have caused a finding that ADER identified as a potential ADR. The system warned clinicians of such confounding diseases in its ADR alerts. [Fig. 1] provides an overview of ADER workflow.

Fig. 1 Complete Adverse Drug Effect Recognizer (ADER) workflow diagram. HPI, history of present illness; PE, physical exam; ROS, review of systems.

Within a few minutes of a clinician creating an H&P note, ADER identified any potential ADRs within the note and generated patient-specific alerts. Each alert displayed the potential ADRs, suspected causative medications, specific evidence extracted from the chart, and any possible confounding diseases. [Fig. 2] shows sample ADER alerts. Physicians could review the alerts in multiple places in Vanderbilt's EHR, “StarPanel”:[51] the EHR Team Summary (used by all care providers); in StarPanel EHR progress note template forms (used to create clinicians' daily progress notes); and, in frequently used, patient-specific, paper-based reports (used by physicians for daily rounds). As part of the current study, alerts included a brief embedded questionnaire to collect feedback from intervention providers.

Fig. 2 Several Adverse Drug Effect Recognizer (ADER) alerts from the study, including potential false positive alerts (renal insufficiency, cough, and edema are potential ADRs, but could also be caused by the patient's kidney failure, chronic obstructive pulmonary disease [COPD], and heart failure).

To prevent alert fatigue,[52] and also at the request of physicians contributing to study design, ADER used noninterruptive alert delivery mechanisms (i.e., its passive displays did not break EHR workflows). The system automatically removed alerts from the EHR when: (1) a provider responded to the alert questionnaire; (2) the patient was discharged; or (3) alert-related questionnaire went unanswered for 5 days.

ADER Evaluation

The ADER intervention study tested the hypothesis that ADER could help providers to better recognize and respond to inpatients' preadmission ongoing ADRs. The study focused on one common class of medications, antihypertensives, which have a myriad of known side effects. The complete set of antihypertensive adverse effects detectable by ADER appears in the Supplementary Material ([Supplementary Tables S1] [S2] [S3], available in the online version). A team of clinical pharmacists reviewed the antihypertensive adverse effects from the previously mentioned DEB2 database, making any necessary clinically relevant revisions prior to initiation of the study. After the target adverse effects had been determined, the authors utilized the aforementioned corpus of 350,000 previous-to-current-study H&Ps to refine KMCI's recognition of those specific effects. This process captured common misspellings, abbreviations, and lexical variants, which were used to improve KMCI recognition of them. To determine the accuracy of concepts recognized by ADER's NLP during the study, the authors analyzed a sample of 100 H&P notes where alerts had been issued. This manual audit attempted to verify that all ADER-identified clinical findings and medications mentioned in alerts were actually present in patient's records.

The Vanderbilt University Institutional Review Board (IRB) approved the study (IRB #141341). Indirectly, the study involved participating house officers' attending physicians and adult inpatients; they did not require informed consent because the intervention simply informed house officers of potential ADRs. The system did not directly change patients' medications. All treatment and medication decisions remained entirely with the care team.

The ADER intervention study began on August 1 and ended on October 31, 2015. The intervention group included consented interns and residents on the General Internal Medicine wards of the Vanderbilt University Hospital. The study length was based on pragmatic convenience—authors were uncertain prior to the study whether the number of alerts that ADER generated would cause alert fatigue. Therefore, consented participating residents and interns, as well as the Director of the Internal Medicine Residency program, agreed to a 3-month trial period as the maximum tolerable duration (in the event that warnings became onerous). The retrospective control group consisted of Internal Medicine interns and residents assigned to the same wards as intervention physicians during a 3-month period just prior to the activation of the system (April 1 through June 30, 2015). To enable comparison of intervention physicians' behaviors to that of controls, the authors ran ADER on the retrospective control group's previous H&P notes, identifying instances when alerts would have been issued had the system been active at the time.

For both intervention and control groups, the study analyzed all clinicians' prescribing behaviors using NLP-based analysis of admission notes, inpatient medication orders, and discharge summaries. This process determined whether preadmission medications from the H&P were continued, held or discontinued during the hospital stay and at discharge. The study compared medication hold/discontinue rates of all intervention clinicians who received alerts and all control physicians who would have received alerts had the system been active. The study also determined separately whether the subgroup of intervention providers who completed the alert-embedded survey (and therefore definitely had seen the alerts) were more likely to change suspected ADR-causing medications compared with all nonalerted control group providers. When evaluating discharge medications, the study only analyzed notes for those patients both admitted and discharged by study subjects (i.e., Internal Medicine house staff). Patients transferred to the care of physicians on another service (e.g., Surgery) were not included in analysis of discharge medications.

When comparing preadmission medications to inpatient medication records, the authors evaluated the difference between intervention and control groups for suspected ADR-causing medications (1) ordered ever during the inpatient stay, (2) ordered in the first 24 hours of the stay, (3) ordered, then stopped or held, as well as (4) suspected ADR-causing medications stopped or held after intervention providers responded to the alert survey versus the overall hold rate in control group. When comparing preadmission medications to discharge summaries, the authors evaluated the difference between intervention and control groups for suspected ADR-causing medications (5) held at discharge and (6) held at discharge for intervention providers who responded to the alert survey. The primary outcome was (1), the difference between groups for suspected ADR-causing medications ordered ever during the inpatient stay.

Whenever intervention subjects' survey responses indicated that they would make or had made changes to suspected ADR-causing medications, the study identified (by analyzing actual orders issued) whether the provider actually followed-through with the medication order changes. An original long version of the survey questionnaire had a low response rate (12%); subjects reported that it was cumbersome and had too many questions. Therefore, a shorter survey ([Fig. 2]) replaced the longer version approximately 3 weeks into the study. Only survey responses obtained during the 9 weeks using the shorter-form survey appear in the results. The authors reviewed open-ended survey comments submitted by intervention subjects and classified them based upon similar themes. Additionally, the authors met with study subjects, including the Chief Resident and Director of the Internal Medicine Residency Program, to discuss their experiences and suggestions at the completion of the intervention.

Statistical Analysis

The study characterized categorical variables using frequencies and proportions and used medians and interquartile range (IQR) to describe continuous variables. To determine whether ADER alerts affected providers' behaviors, authors analyzed medication order changes during the inpatient stay and at discharge. The analysis compared the rates of medication changes between the control and intervention groups using chi-square tests with a significance level of 0.05. When performing secondary analyses on data, the authors used Bonferroni correction to adjust the significance level for all comparisons to account for multiple testing. All statistical analyses used R version 2.15.2.

Results

[Table 1] illustrates ADER study results for both the intervention and retrospective control groups. The 3-month preintervention control group included 2,666 H&P notes written by 138 interns and residents. During the 3-month intervention period, the ADER scanned 2,312 H&P notes written by 137 interns and residents. No significant differences occurred between the intervention and control groups for any metrics shown in [Table 1] (p > 0.05). The ADER system detected potential ADRs in 30 to 32% of both groups' notes, with 78 to 79% triggered by H&P text findings and 21 to 22% by laboratory results. The median time from when an intervention group physician saved an admission note to the EHR to when ADER posted an alert was 8 minutes (IQR = 6–9 minutes). The median time between ADER issuing an alert and an intervention subject responding to the embedded survey questions was 24 hours (IQR = 17–37 hours).

Table 1 ADER study results for intervention and control groups

	H&Ps	Unique patients		Notes with potential ADRs		Average number of antihypertensive medications per note
	N	N	%	N	%	Notes without potential ADRs		Notes with potential ADRs
Intervention	2,312	2,049	89	701	30	0.73		2.4
Control	2,666	2,352	88	867	32	0.76		2.4
	Potential ADRs per alert	Adverse effect source		Number of potential ADRs found per alert
		H&P text	Labs	Min.	25%	Median	75%	Max
Intervention	2.3	79	21	1	1	2	3	15
Control	2.4	78	22	1	1	2	3	23
	Admissions with discharge note		Unique patients with discharge note		Average length of stay	Number of providers
	N	%	N	%		Total	With discharge
Intervention	1,835	79	1,646	90	115 h	137	137
Control	2,160	81	1,916	89	112 h	138	128

Abbreviations: ADER, Adverse Drug Effect Recognizer; ADR, adverse drug reaction; H&P, history and physical examination.

ADER NLP Accuracy

Manual expert review of 100 intervention group H&P notes for which ADER had issued potential ADR alerts revealed that 98% of the 157 medications that ADER identified as current therapies were correct. In three instances, ADER incorrectly identified drugs that the H&Ps described as previously discontinued. Of 136 total adverse effects (findings or diseases) that ADER identified from H&P notes, 116 (86%) were confirmed as “present” in the chart. Of the 20 incorrectly identified adverse effects, 15 were mentioned as being absent but were not recognized as “negated” by KMCI due to use of nonstandard wording. The remaining misidentified concepts were mentioned only as possible points of differential diagnosis or past inactive diagnoses. Of the 30 clinical findings the ADER recognized from laboratory results, 29 (97%) were accurate. The only incorrectly classified numerical laboratory result was “hyperkalemia” that had been noted in comments to be due to a hemolyzed sample.

Comparing Admission Medications to Inpatient Medication Orders

The intervention group ordered significantly fewer suspected ADR-causing preadmission medications at any point during the inpatient stay compared with the control group (47% vs. 58%, p < 0.001). Similarly, during the first 24 hours after admission, intervention physicians ordered only 28% of 1,140 suspected ADR-causing preadmission medications flagged by ADER; the control group had ordered 39% of 1,468 suspected ADR-causing preadmission medications during the same interval (28% vs. 39%, p < 0.001). See [Table 2].

Table 2 Preadmission medications compared with ordered inpatient medications, testing the null hypothesis that the proportion for the intervention group was equal to the proportion for the control group

	Suspected ADR-causing medications	Ordered ever during inpatient stay		Ordered in first 24 hours		Ordered, then stopped or held
		N	Proportion	N	Proportion	N	Proportion
Intervention	1,140	532	0.467	315	0.276	91	0.171
Control	1,468	846	0.576	570	0.39	125	0.148
Difference (95% CI) p-value			–0.109 (–0.149, –0.070) p < 0.001		–0.114 (–0.149, –0.075) p < 0.001		0.023 (–0.018, 0.065) p = 0.28

Abbreviations: ADR, adverse drug reaction; CI, confidence interval.

Among suspected ADR-causing medications ordered during the inpatient stay, 84% (711/846) were ordered within 24 hours of admission in the control group; in the intervention group, 77% (410/532) were ordered within 24 hours of admission (p = 0.002). Suspected ADR-causing medications were stopped before patient discharge at rates of 14 and 16% for the control and intervention groups, respectively (p = 0.40). Among medications started after 24 hours into the inpatient stay, subsequent hold rates were 19 and 20% in the control and intervention groups, respectively (p = 1.00).

Among suspected ADR-causing preadmission medications ordered at any point in the inpatient stay, the rate at which they were subsequently held or discontinued did not vary significantly between groups (see [Table 2], p = 0.28). However, secondary analysis, as shown in [Table 3], revealed that the subset of intervention providers who responded to alert-embedded surveys—and therefore who definitely viewed the alerts—subsequently held or discontinued 25% (41/167) of the then-active suspected ADR-causing preadmission medications. This rate is significantly different than the aforementioned control group rate (25% vs. 15%, p = 0.003). In the intervention group, the median time between responding to the survey and subsequent medication changes was 25 hours (IQR = 13–66).

Table 3 Preadmission medications ordering during inpatient stay later stopped or held, testing the null hypothesis that the proportion for the intervention group after survey response was equal to the overall proportion for the control group

	Suspected ADR-causing meds	N		N	Proportion
Intervention	Active order at time of ADER alert survey response	167	Stopped or held after survey response	41	0.246
Control	Ordered ever during the inpatient stay	846	Ordered ever, then stopped or held	125	0.148
Difference (95% CI) p-value					0.098 (0.025, 0.171) p = 0.003

Abbreviations: ADER, Adverse Drug Effect Recognizer; ADR, adverse drug reaction; CI, confidence interval.

Comparing Admission Medications to Discharge Medications

Approximately 20% of intervention and control group discharge summaries were not generated by study subjects and thus were excluded from review. [Table 4] shows hold/discontinuation rates of suspected ADR-causing preadmission medications at discharge for the remaining 80% of discharge summaries.

Table 4 Preadmission medications compared with discharge medications testing the null hypothesis that the proportion for the intervention group was equal to the proportion for the control group

	Suspected ADR-causing preadmission medications: full intervention group versus full control group					Suspected ADR-causing preadmission medications; intervention group survey respondents versus full control group
	Total	Medication missing from discharge note		Held or discontinued at discharge (explicitly)		Total	Held or discontinued at discharge (explicitly)
	N	N	Proportion	N	Proportion	N	N	Proportion
Intervention	927	104	0.112	184	0.198	258	97	0.376
Control	1,225	75	0.061	287	0.234	1,225	287	0.234
Difference (95% CI) p-value			0.051 (0.025, 0.076) p < 0.001		–0.036 (–0.072, 0.0001) p = 0.053			0.142 (0.076, 0.208) p < 0.001

Abbreviations: ADR, adverse drug reaction; CI, confidence interval.

When discharge notes did not mention previously alerted preadmission medications, the study could not definitively determine whether the medications had been held or stopped at discharge (104/927 = 11% from intervention, 75/1,225 = 6.1% from control, p < 0.001). Actual specific mentions in discharge notes of holds/discontinuations for alerted medications did not differ significantly: 20% (184/927) in the intervention group and 23% (287/1225) in the control group (p = 0.053). However, intervention providers who responded to the alert-embedded survey held or discontinued 38% (97/258) of suspected ADR-causing medications at discharge, significantly different (p < 0.001) than the overall 23% rate for the control group.

Survey Questionnaire Results and Provider Comments

Intervention subjects responded to 53% (298/560) of the short-version alert-embedded questionnaires (see [Table 5]). For survey respondents who stated that they would change the patient's therapy due to identified ADRs, subsequent medication holds or dosage decreases occurred more than 90% of the time. For the 38 intervention subjects who responded that they were uncertain whether they would make a change, analysis identified 24 instances (63%) where one or more suspected ADR-causing medications was subsequently changed. Nearly three-quarters of survey responses indicated alerts either somewhat or fully merited consideration for their patients. More than half said alerts were helpful or possibly helpful in managing the patient.

Table 5 Responses to ADER alert survey questionnaires (n = 298)

Question	Response
		N	(%)		N	(%)		N	(%)
Did any of these alerts merit at least passing consideration for this patient?	Yes	144	(48)	Somewhat	74	(24)	No	80	(26)
Were any of these alerts helpful in managing this patient?	Yes	87	(29)	Possibly	80	(26)	No	131	(43)
Will you (or did you already) change patient's therapy due to these possible ADRs?	Yes	91	(30)	Uncertain	38	(12)	No	169	(56)

Abbreviations: ADER, Adverse Drug Effect Recognizer; ADR, adverse drug reaction.

Approximately 30% of the 52 optional free-text survey comments indicated that ADER was correct in identifying an ADR. Conversely, 30% of the survey comments stated that claimed adverse effects were more likely due the patient's underlying medical conditions (usually those recognized by ADER as possible confounders). For example, when ADER identified renal insufficiency or acute kidney injury as an ADR, providers often indicated that the patient's existing end-stage renal disease was the more likely cause of impaired renal function.

In ∼20% of survey comments, providers indicated that the medications were held, but not due to the ADER-recognized ADRs. In at least two cases, however, the H&P Plan section indicated that a provider had in fact held the medications for the reasons identified by ADER. In 10% of the comments, providers indicated that the patients were not suffering from the suspected ADR. In one of those cases, ADER had recognized “gout” as a possible ADR. The patient had a previous diagnosis of gout, but no acute gout exacerbation was present on admission. In another case, ADER “incorrectly” identified that the patient had a cough because of the phrase, “Hurts with movement, cough, sneezing, position changes.” Several survey respondents suggested that they had already recognized the ADR suggested by ADER.

Study subjects also provided feedback to the authors directly. Participating providers generally viewed ADER favorably, but suggested that ADER might be more useful for those medications that internists prescribe less frequently than antihypertensives, such as psychiatric medications. They also expressed that potential confounders included in ADER alerts were helpful in assessing the veracity of suspected ADRs and in considering whether medications were possibly exacerbating the recognized findings.

Medication Holds Analyzed by ADR

Among the 940 intervention group alerts, 41 were distinct adverse effects (patient findings/disorders). The most frequent alert-related conditions (occurring more than 50 times) were: renal insufficiency, light headedness, edema, dizziness, hypotension, and syncope. Suspected causative medications were held at discharge at rates of 35% for hypotension, 28% for renal insufficiency, 42% for hyperkalemia, 36% for hypokalemia, and 50% for hyponatremia. The Supplementary Material ([Supplementary Tables S1] [S2] [S3], available in the online version) provides hold/discontinue rates for all medications with detected ADRs. The online Supplementary Material also reports comparisons of hold rates between suspected ADR-causing medications and those not suspected of causing ADRs.

Discussion

The current study illustrates the potential for automated alerting systems like ADER to help providers identify unrecognized ADRs. Physicians receiving ADER alerts held or discontinued more suspected ADR-causing medications at multiple points in the patient care workflow compared with physicians who did not receive alerts. In survey responses, physicians stated that ADER alerts more than half the time helped with patient management. Analysis of ADER's accuracy revealed that the majority EHR-extracted findings and medications were correct, but also revealed opportunities for improvements in NLP, particularly in regards to negation detection.

Intervention providers receiving ADER alerts ordered significantly fewer suspected ADR-causing preadmission medications in the first 24 hours after admission and at any time during the inpatient stay. As a whole, intervention providers receiving alerts were no more likely to discontinue suspected ADR-causing preadmission medications at discharge than the control group (although they had ordered fewer such medications overall at admission). Nevertheless, providers who responded to the alert-embedded survey were more likely to hold or discontinue ADR-causing medications during the patient's hospitalization and at discharge. Two explanations are possible—either providers who believed that a potential ADR alert was valid were more likely to respond to the survey, or providers who believed ADER alerts were true more often addressed potential ADR-causing medications at discharge. Further study could determine whether ADER alerts would be more effective if they interrupted workflows, since the current study used noninterruptive alerts. Why intervention physicians in some instances ordered previously alerted ADR-causing medications later during an admission is also a topic for further research. A possible explanation is that when ADR-related findings resolved, re-starting the medication was appropriate (e.g., hypotension in patients with baseline hypertension).

From a technical standpoint, the ADER approach is generalizable to many inpatient facilities. The system only requires that a hospital have an EHR system that can electronically export (under appropriate security) admission H&P notes and laboratory values as they are generated to a generic desktop-class computer running the ADER software. Another key requirement is the hospital EHR's ability to incorporate ADER-generated alerts into the EHR workflow in locations likely to be viewed by clinicians.

Comparison with Prior Research

Use of NLP to identify potential ongoing ADRs from outpatient medications at the time of admission is novel. A substantial body of work exists regarding other methods of ADR detection and associated determinations of physicians' responses. In 2017, Dexheimer et al reported that providers in a pediatric hospital, after receiving inpatient dosing, allergy, and drug–drug interaction alerts cancelled or modified medication orders an average of 12.6, 7.0, and 16.6%, respectively.[53] A 2003 study of five adult primary care practices found that physicians accepted only 8.8% of drug allergy and 10.6% of high-severity drug–drug interaction alerts.[54] Further analysis revealed physicians were more likely to override alerts for renewals than new prescriptions and that house officers were less likely to prescribe alerted medications than other physicians. A different study suggested that house staff were more likely to override ADR alerts than staff physicians.[55] In a classic 1976 study, McDonald evaluated provider responses to electronic reminders, including suggestions to observe a physical finding or ask about a specific symptom, order a diagnostic study or change or start a therapeutic regimen.[56] The study found physicians responded to 51% of events when given electronic reminders compared with 22% of events when they did not receive reminders. Of note, few such reminders involved medication changes due to potential ADRs. The study indicated that physicians comply with a higher proportion of electronic recommendations when protocols are more precise.

The current study's physician alert response rates exceed the aforementioned previous studies regarding other types of ADR alerts. A possible explanation is that the current study named in its alerts the suspected ADR symptoms that were present in the patient at admission. The ADER methodology is significantly different than that of the above-mentioned efforts, as well as the adverse effect detection studies discussed in the background. The previous efforts utilized sudden discontinuations of medications, administrations of known antidotes, and ADR-related billing codes, all of which indicate that providers had already recognized the ADRs. They also warned about dosing errors, allergies, and possible drug–drug interactions, which often do not take into account the actual symptoms being experienced by the patient. The ADER system is novel in that it utilizes NLP of provider-generated admission notes to warn clinicians about potential ongoing ADRs, as evidenced by documented symptoms and findings, due to outpatient medications. It has the potential to detect unrecognized ADRs and provide a safeguard to ensure all providers are aware of the ADRs their patients may be experiencing.

Limitations and Future Work

Several limitations pertain to the study. The intervention only included Internal Medicine interns and residents as alert recipients. That survey responses followed alerts by ∼24 hours suggests that house officers may have waited to discuss alerts with the ward team and attending physicians before acting. Additionally, house staff often rotate “off service” so that the physicians writing discharge orders may not have been the ones who received the initial ADER alerts at the time of admission, even though the discharging physician qualified as study participants by membership in the Internal Medicine house staff. A wider range of subjects (e.g., academic attending physicians, community-based physicians, nurse practitioners, physician assistants) should be evaluated in teaching and nonteaching settings. The study limited the ADER intervention to antihypertensive medications so that potential alert fatigue would not confound initial evaluation of ADER's utility—as might occur if all categories of medications generated alerts. The study targeted the commonly prescribed antihypertensive class because they cause a wide variety of ADRs. Admittedly, providers who regularly prescribe these medications are more likely to be aware of their potential ADRs, and study participants suggested that ADER might be more useful for medications they prescribe less frequently. In the future, ADER should be expanded, in consultation with clinical experts, to cover additional drug classes and their related ADRs, with particular focus on uncommon or severe ADRs.

The study analyzed 3 months of intervention data compared with a similar-duration retrospective control group. While analysis showed the control group to be comparable to the intervention group in nearly all respects, the groups contained different participants and the admissions occurred at somewhat different times in the academic year (controls were soon-to-be promoted house officers; intervention subjects had arrived in their current positions 1 month prior to study initiation). A randomized controlled trial involving multiple institutions and all drug classes could provide a more definitive result than the current study. Before undertaking such a study, the potentially correctable flaws in ADER's NLP methods (identified in this study) should be addressed.

To evaluate the potential of this methodology, this study focused only on the accuracy of the ADER alerts that were presented. Future studies with additional drug classes should consider both precision and recall to determine which potential ADRs ADER misses (i.e., fails to generate alerts when appropriate to warn clinicians). To reduce alert fatigue, more advanced confounder detection should be studied with the goal of filtering out false-positive alerts when potential ADRs have a significantly more likely clinical cause. Future studies will also include more in-depth qualitative analysis of providers' impressions and resultant behavior.

Conclusion

Intervention subjects reduced ordering of suspected ADR-causing medications at multiple points during the admission compared with the control group. Systems like ADER have the potential to improve both recognition of adverse effects and discontinuation of medications causing preadmission ADRs. Future ADER-like systems must supply relevant information to care providers at the right place and time to improve patient safety, potentially decrease cost, and improve the quality of care.

Clinical Relevance Statement

This study shows that using NLP extraction of patient preadmission medications and findings to generate real-time ADR alerts can affect provider ordering behavior. This approach could be generalized to other facilities that use electronic health records. Clinical decision support systems like ADER could help to improve the safety of care delivery in the future.

Multiple Choice Questions

1. Approximately what percentage of ADER-identified adverse drug reactions were found in the text of the admission note, as compared with solely derived from laboratory test results?

   • 20%

   • 40%

   • 60%

   • 80%

     Correct Answer: The correct answer is option d. Nearly 80% of suspected ADRs identified by ADER (79% in the intervention group and 78% in the control group) were identified from NLP on the text of the admission note. Only 20% of antihypertensive ADRs detected by the system were identifiable from laboratory test results alone.

2. How often did intervention clinicians receiving an alert order the potentially offending medication during the hospital stay, and how often did control physicians do so?

   • 21% versus 25%

   • 47% versus 37%

   • 47% versus 58%

   • 67% versus 68%

     Correct Answer: The correct answer is option c. While intervention clinicians receiving an alert ordered suspected ADR-causing medications 47% of the time, physicians in the control group ordered the potentially offending medications 58% of the time (p < 0.01).

Conflict of Interest

None.

Acknowledgments

The authors acknowledge the contributions of several individuals to this study. Vanderbilt pharmacists Mark Sullivan, Terry Bosen, David DiPersio, Fred Hargrove, and Amy Myers helped review and refine ADRs alerted by the ADER system. Dario Giuse, Yun Wang, Jun Kunavut, Wing Liu, Joe Warner, Zhao Zuo, and Peter Speltz provided technical assistance in integrating ADER into the Vanderbilt EHR and order entry systems. John McPherson, the director of the Vanderbilt Internal Medicine Residency Program, and Thomas Atwater, Chief Resident in Internal Medicine, participated in planning and implementing the intervention study at Vanderbilt University Hospital. The authors also thank the Vanderbilt Internal Medicine residents and interns who participated in the study.

Authors' Contributions

All authors contributed to the study design. J.C.S. and R.A.M. wrote the study protocols, acquired IRB approval, and recruited participants. J.C.S. performed all software development, debugging, data acquisition, and data processing under the supervision of R.A.M. With the assistance of K.B.J., R.A.M., and technical staff (see Acknowledgments), J.C.S. integrated the ADER system into the EHR and maintained the system during the study. J.C.S. and Q.C. performed the statistical analysis. J.C.S. and R.A.M. interpreted the results and wrote the first drafts of the manuscript. Q.C., J.C.D., D.M.R., and K.B.J. provided critical comments, suggestions, and changes to the manuscript. All authors approved the final manuscript.

Protection of Human and Animal Subjects

This study was performed in compliance with all applicable ethical standards for medical research involving human subjects. The Vanderbilt University Institutional Review Board approved this study (IRB #141341).

• References

• 1 U.S. Department of Health and Human Services, Office of Disease Prevention and Health Promotion. National Action Plan for Adverse Drug Event Prevention. [Internet]. Washington, DC; 2014. Available at: https://health.gov/hcq/ade-action-plan.asp . Accessed September 24, 2017
• 2 CDC-Basic-Medication Safety Program. Available at: http://www.cdc.gov/medicationsafety/basics.html . Accessed October 3, 2017
• 3 Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies. JAMA 1998; 279 (15) 1200-1205
• 4 Pirmohamed M, James S, Meakin S. , et al. Adverse drug reactions as cause of admission to hospital: prospective analysis of 18 820 patients. BMJ 2004; 329 (7456): 15-19
• 5 Kongkaew C, Noyce PR, Ashcroft DM. Hospital admissions associated with adverse drug reactions: a systematic review of prospective observational studies. Ann Pharmacother 2008; 42 (07) 1017-1025
• 6 Atiqi R, van Bommel E, Cleophas TJ, Zwinderman AH. Prevalence of iatrogenic admissions to the Departments of Medicine/Cardiology/ Pulmonology in a 1,250 bed general hospital. Int J Clin Pharmacol Ther 2010; 48 (08) 517-524
• 7 Hakkarainen KM, Gyllensten H, Jönsson AK, Andersson Sundell K, Petzold M, Hägg S. Prevalence, nature and potential preventability of adverse drug events - A population-based medical record study of 4970 adults. Br J Clin Pharmacol 2014; 78 (01) 170-183
• 8 International drug monitoring: the role of national centres. Report of a WHO meeting. World Health Organ Tech Rep Ser 1972; 498: 1-25
• 9 Edwards IR, Aronson JK. Adverse drug reactions: definitions, diagnosis, and management. Lancet 2000; 356 (9237): 1255-1259
• 10 Stausberg J. International prevalence of adverse drug events in hospitals: an analysis of routine data from England, Germany, and the USA. BMC Health Serv Res 2014; 14: 125
• 11 Pedrós C, Quintana B, Rebolledo M, Porta N, Vallano A, Arnau JM. Prevalence, risk factors and main features of adverse drug reactions leading to hospital admission. Eur J Clin Pharmacol 2014; 70 (03) 361-367
• 12 Nebeker JR, Barach P, Samore MH. Clarifying adverse drug events: a clinician's guide to terminology, documentation, and reporting. Ann Intern Med 2004; 140 (10) 795-801
• 13 Classen DC, Pestotnik SL, Evans RS, Lloyd JF, Burke JP. Adverse drug events in hospitalized patients. Excess length of stay, extra costs, and attributable mortality. JAMA 1997; 277 (04) 301-306
• 14 Reducing and Preventing Adverse Drug Events To Decrease Hospital Costs | Agency for Healthcare Research & Quality (AHRQ). Available at: http://www.ahrq.gov/research/findings/factsheets/errors-safety/aderia/index.html
• 15 Sultana J, Cutroneo P, Trifirò G. Clinical and economic burden of adverse drug reactions. J Pharmacol Pharmacother 2013; 4 (05) (Suppl. 01) S73-S77
• 16 Lahue BJ, Pyenson B, Iwasaki K, Blumen HE, Forray S, Rothschild JM. National burden of preventable adverse drug events associated with inpatient injectable medications: healthcare and medical professional liability costs. Am Health Drug Benefits 2012; 5 (07) 1-10
• 17 Handler SM, Altman RL, Perera S. , et al. A systematic review of the performance characteristics of clinical event monitor signals used to detect adverse drug events in the hospital setting. J Am Med Inform Assoc 2007; 14 (04) 451-458
• 18 Murff HJ, Patel VL, Hripcsak G, Bates DW. Detecting adverse events for patient safety research: a review of current methodologies. J Biomed Inform 2003; 36 (1-2): 131-143
• 19 Classen DC, Pestotnik SL, Evans RS, Burke JP. Computerized surveillance of adverse drug events in hospital patients. JAMA 1991; 266 (20) 2847-2851
• 20 Evans RS, Pestotnik SL, Classen DC. , et al. Development of a computerized adverse drug event monitor. Proc Annu Symp Comput Appl Sic Med Care Symp Comput Appl Med Care; 1991:23–27
• 21 Evans RS, Pestotnik SL, Classen DC, Bass SB, Burke JP. Prevention of adverse drug events through computerized surveillance. Proc Annu Symp Comput Appl Sic Med Care Symp Comput Appl Med Care; 1992:437–441
• 22 Jha AK, Kuperman GJ, Teich JM. , et al. Identifying adverse drug events: development of a computer-based monitor and comparison with chart review and stimulated voluntary report. J Am Med Inform Assoc 1998; 5 (03) 305-314
• 23 Honigman B, Lee J, Rothschild J. , et al. Using computerized data to identify adverse drug events in outpatients. J Am Med Inform Assoc 2001; 8 (03) 254-266
• 24 Bates DW, Evans RS, Murff H, Stetson PD, Pizziferri L, Hripcsak G. Detecting adverse events using information technology. J Am Med Inform Assoc 2003; 10 (02) 115-128
• 25 Murff HJ, Forster AJ, Peterson JF, Fiskio JM, Heiman HL, Bates DW. Electronically screening discharge summaries for adverse medical events. J Am Med Inform Assoc 2003; 10 (04) 339-350
• 26 Raschke RA, Gollihare B, Wunderlich TA. , et al. A computer alert system to prevent injury from adverse drug events: development and evaluation in a community teaching hospital. JAMA 1998; 280 (15) 1317-1320
• 27 Peterson JF. Drug-lab triggers have potential to prevent adverse drug events in outpatients. J Am Med Inform Assoc 2002; 9 (90061): 39-40
• 28 Chen ES, Hripcsak G, Xu H, Markatou M, Friedman C. Automated acquisition of disease drug knowledge from biomedical and clinical documents: an initial study. J Am Med Inform Assoc 2008; 15 (01) 87-98
• 29 Wang X, Hripcsak G, Markatou M, Friedman C. Active computerized pharmacovigilance using natural language processing, statistics, and electronic health records: a feasibility study. J Am Med Inform Assoc 2009; 16 (03) 328-337
• 30 Wang X, Chase H, Markatou M, Hripcsak G, Friedman C. Selecting information in electronic health records for knowledge acquisition. J Biomed Inform 2010; 43 (04) 595-601
• 31 Wang X, Chase HS, Li J, Hripcsak G, Friedman C. Integrating heterogeneous knowledge sources to acquire executable drug-related knowledge. AMIA Annu Symp Proc AMIA Symp 2010;2010:852–856
• 32 Li Y, Salmasian H, Harpaz R, Chase H, Friedman C. Determining the Reasons for Medication Prescriptions in the EHR using Knowledge and Natural Language Processing. AMIA Annu Symp Proc AMIA Symp 2011;2011:768–776
• 33 LePendu P, Iyer SV, Bauer-Mehren A. , et al. Pharmacovigilance using clinical notes. Clin Pharmacol Ther 2013; 93 (06) 547-555
• 34 Harpaz R, Vilar S, Dumouchel W. , et al. Combing signals from spontaneous reports and electronic health records for detection of adverse drug reactions. J Am Med Inform Assoc 2013; 20 (03) 413-419
• 35 Harpaz R, DuMouchel W, Shah NH, Madigan D, Ryan P, Friedman C. Novel data-mining methodologies for adverse drug event discovery and analysis. Clin Pharmacol Ther 2012; 91 (06) 1010-1021
• 36 Tatonetti NP, Ye PP, Daneshjou R, Altman RB. Data-driven prediction of drug effects and interactions. Sci Transl Med 2012; 4 (125) 125ra31
• 37 Harpaz R, Callahan A, Tamang S. , et al. Text mining for adverse drug events: the promise, challenges, and state of the art. Drug Saf 2014; 37 (10) 777-790
• 38 Duke JD, Friedlin J. ADESSA: A Real-Time Decision Support Service for Delivery of Semantically Coded Adverse Drug Event Data. AMIA Annu Symp Proc AMIA Symp 2010;2010:177–181
• 39 Warrer P, Hansen EH, Juhl-Jensen L, Aagaard L. Using text-mining techniques in electronic patient records to identify ADRs from medicine use. Br J Clin Pharmacol 2012; 73 (05) 674-684
• 40 Denny JC, Miller RA, Johnson KB, Spickard III A. Development and evaluation of a clinical note section header terminology. AMIA Annu Symp Proc AMIA Symp 2008:156–160
• 41 Denny JC, Smithers JD, Miller RA, Spickard III A. “Understanding” medical school curriculum content using KnowledgeMap. J Am Med Inform Assoc 2003; 10 (04) 351-362
• 42 Denny JC, Peterson JF, Choma NN. , et al. Extracting timing and status descriptors for colonoscopy testing from electronic medical records. J Am Med Inform Assoc 2010; 17 (04) 383-388
• 43 MetaMap - A Tool for Recognizing UMLS Concepts in Text. Available at: https://metamap.nlm.nih.gov/
• 44 Xu H, Stenner SP, Doan S, Johnson KB, Waitman LR, Denny JC. MedEx: a medication information extraction system for clinical narratives. J Am Med Inform Assoc 2010; 17 (01) 19-24
• 45 Peterson JF, Shi Y, Denny JC. , et al. Prevalence and Clinical Significance of Discrepancies within Three Computerized Pre-Admission Medication Lists. AMIA Annu Symp Proc AMIA Symp 2010; 2010:642–646
• 46 Unified Medical Language System (UMLS). Available at: https://www.nlm.nih.gov/research/umls/ . Accessed September 24, 2017
• 47 RxNorm Overview. Available at: https://www.nlm.nih.gov/research/umls/rxnorm/overview.html . Accessed September 24, 2017
• 48 SNOMED CT. Available at: https://www.nlm.nih.gov/healthit/snomedct/ . Accessed September 24, 2017
• 49 Chapman WW, Bridewell W, Hanbury P, Cooper GF, Buchanan BG. A simple algorithm for identifying negated findings and diseases in discharge summaries. J Biomed Inform 2001; 34 (05) 301-310
• 50 Smith JC, Denny JC, Chen Q. , et al. Lessons learned from developing a drug evidence base to support pharmacovigilance. Appl Clin Inform 2013; 4 (04) 596-617
• 51 Giuse DA. Supporting communication in an integrated patient record system. AMIA Annu Symp Proc AMIA Symp. 2003:1065
• 52 Phansalkar S, van der Sijs H, Tucker AD. , et al. Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc 2013; 20 (03) 489-493
• 53 Dexheimer JW, Kirkendall ES, Kouril M. , et al. The effects of medication alerts on prescriber response in a pediatric hospital. Appl Clin Inform 2017; 8 (02) 491-501
• 54 Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003; 163 (21) 2625-2631
• 55 Cho I, Slight SP, Nanji KC. , et al. The effect of provider characteristics on the responses to medication-related decision support alerts. Int J Med Inform 2015; 84 (09) 630-639
• 56 McDonald CJ. Protocol-based computer reminders, the quality of care and the non-perfectability of man. N Engl J Med 1976; 295 (24) 1351-1355