Keywords
system improvement - disease management - clinical research informatics - allergy and immunology - clinical data management
Background and Significance
Background and Significance
Asthma exacerbation, characterized by an acute worsening of asthma symptoms, is quite prevalent, with 1.6 million emergency department (ED) visits a year in the United States.[1]
[2] These are associated with significant morbidity in terms of abrupt decline in physical function, lung function, and health-related quality of life, including missed school and work days, increased health care utilization, and mortality.[3]
[4]
[5]
[6]
[7]
[8] Multiple predictors for future risk of asthma exacerbation have been found but recent exacerbation history, especially leading to ED visit or hospitalization, is the strongest predictor of future asthma exacerbations.[9]
[10]
[11] Evidence suggests that currently available therapy prevents exacerbation during treatment.[12]
[13] Therefore, asthma exacerbation leading to ED visit is a potentially preventable clinical outcome and is of great interest both to health services research as an indicator of quality of care to the patient, and to epidemiologic studies.
It has been previously shown that less than a fifth of the asthma population was responsible for more than four-fifths of the total direct costs emphasizing the need for better management of a high-risk cohort.[14] For the purposes of quality improvement, we need performance measures that are easily identified using clear administrative or clinical (electronic medical record [EMR]) criteria, actionable based on evidence-based guidelines without ambiguity for most patients, and reliable. Commonly used Healthcare Effectiveness Data and Information Set asthma quality metrics pertain to controller medication filling and controller-to-total asthma medication ratio.[15] However, the use and response to asthma medication is variable unlike other chronic diseases and these process measures are inadequate to identify the risk of health care utilization.[16] Asthma exacerbation leading to ED visit is a good performance metric in the asthma population as it is the outcome that needs to be identified, managed, and avoided for the previously mentioned reasons. As performance measures are definitive standards of care against which the care provided is judged, extensive care needs to be taken in defining such a measure.[17]
Identification of asthma-related ED visits in studies has been conventionally done by International Classification of Diseases, Ninth Revision (ICD-9) discharge codes, chief complaints, and medication use in EMR and claims data.[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25]
[26]
[27]
[28]
[29] A systematic review of validated methods to capture acute bronchospasm using administrative or claims data showed that there are only two studies with validation characteristics among the 38 studies using an algorithm.[30] An EMR algorithm based on chief complaints implemented in the Southern U.S. showed sensitivity of 45%, specificity of 92%, positive predictive value (PPV) of 79%, and negative predictive value (NPV) of 70%.[31] A study based on claims data showed that claims failed to identify 29% of encounters with asthma diagnoses and 45% of nebulization procedures administered during encounters; 30% of documented asthma prescriptions were not associated with filed claims and vice versa.[32] Another EMR study performed in the Midwest since the systematic review, based on Bayesian network which used electronic information available at the time of ED visit like age, respiratory rate, chief complaint, oxygen saturation, and acuity level, and historical data like past medical history, medications, and billing codes, yielded a PPV of 69.9%.[33] Severe chronic lower respiratory disease exacerbations which combined asthma and other respiratory diseases as measured by discharge diagnosis code had a predictive value of 85 to 95% based on the threshold for gold standard.[34]
In spite of the lack of data to correctly identify ED visit related to asthma exacerbation, the above conventional methods are used by health systems to evaluate their performance using EMR data, by health plans to track utilization using claims data, and by researchers to assess outcome in epidemiologic studies using either data.[35]
[36]
[37]
[38]
[39] Routine operation of health care systems provides tremendous electronic information at patient level, but this needs to be harnessed and adapted for research and administrative purposes. Developing algorithms using clinical and administrative data elements help to identify various health outcomes of interest. Given the need for the use of asthma exacerbation-related ED visit as a performance measure for systems-based practice and for epidemiologic studies based on health care databases, there is an urgent need to develop an algorithm with adequate performance metrics, based on any available data source—EMR or claims.
Objective
We propose to utilize multiple data elements available in the EMR and claims database to create separate algorithms with high validity for clinical and research purposes to identify asthma exacerbation-related ED visit among the general population.
Methods
Study Overview
This is a retrospective study of 1,000 ED visits randomly selected out of all eligible visits, for patients in a single health system in Pennsylvania with asthma, with a subset of these visits also having claims data. Chart review was performed to classify ED visits into those related to asthma exacerbation or otherwise, which was the gold standard. Different data elements, like demographics, chief complaints, vital signs, medications, and discharge diagnoses as available in different databases, were obtained to create separate asthma exacerbation-related ED visit algorithms in EMR and claims by comparing against the gold standard.
Study Population and Subject Selection
Geisinger is a large integrated health system which serves more than 400,000 primary care patients across a 45-county area in Pennsylvania, United States, with this population being representative of the general population of central and northeastern Pennsylvania.[40] It utilizes EMR software by Epic Systems Corporation. Geisinger Health Plan (GHP) is a full-service regional plan covering approximately 500,000 people.[41] Our inclusion criteria were ED visit to a Geisinger ED from January 1, 2006, to October 28, 2013, of patients who were of age 4 to 40 years with asthma in the problem list (ICD-9 code 493.xx). Exclusion criteria were chronic obstructive pulmonary disease (COPD) (ICD-9 codes 491, 492, 496) or coronary artery disease/heart failure (ICD-9 codes 410–414, 428) or cystic fibrosis (ICD-9 code 277.xx) or bronchiectasis (ICD-9 code 494) in the problem list or ED visits resulting in hospital admission. These were comorbidities which can present with lower respiratory symptoms, making it difficult to attribute an exacerbation to one specific disease condition.
Given that not all patients who visit the Geisinger ED have GHP insurance, we enriched our study sample for ED visits with GHP insurance, providing claims data. We also enriched for ED visits with classifiers of interest, as asthma patients visit ED for asthma-related and nonasthma-related concerns. A total of 4,708 eligible ED visits with both EMR and claims data were identified and we aimed for a total of 1,000 ED visits. The visits were stratified into 15 strata according to the presence or absence of classifiers of interest (primary diagnosis, secondary diagnosis, respiratory complaint, bronchodilator use) in isolation or in combination. First, from these strata, random sample of visits of predetermined proportion (as determined by the size of the stratum in relation to other strata and the probable value of that stratum in predicting asthma exacerbation), or all visits, if the strata were small, were taken for a total n = 780 ED visits. An additional 220 ED visits were randomly identified from similar strata in the eligible 20,379 ED visits in the Geisinger population without claims data, for a total of 1,000 ED visits. This stratified sampling methodology was done to increase the probability of presence of classifiers of interest in the study data set.
Gold Standard Generation
The research team of allergist, informaticist, and primary care physicians developed a structured chart review abstraction tool by consensus decision-making, which evaluated both the exclusion criteria and the diagnosis of asthma exacerbation in the ED visit. The abstraction tool included various data elements from problem list, past medical history, history of present illness, physical exam, assessment and plan in the ED notes, medication administration, and progress notes, and was used to conclude the presence or absence of asthma exacerbation during ED visit. Two reviewers were trained on chart abstraction and performed chart reviews. A subset of charts was randomly selected from the study sample for validation of the exacerbation classification by another independent reviewer. The asthma exacerbation status in the ED visit by chart review was considered as the gold standard in algorithm development.
Definitions of Predictor Variables
Principal and secondary diagnoses of asthma exacerbation were identified by clinically coded ICD-9 code of 493.xx as listed in the encounter or claims. If an ED visit had asthma as both principal and secondary diagnosis, it was only counted for principal diagnosis. Respiratory complaints were identified by the terms “upper respiratory infection,” “cough,” “flu,” “congestion,” “wheezing,” “respiratory distress,” “short of breath,” “asthma,” “asthma attack,” “chest discomfort,” “chest pressure,” “pneumonia,” “chest pain,” “bronchitis,” “hyperventilating,” “hemoptysis,” “cold symptoms,” “airway obstruction,” or “chest tightness” listed in the chief complaint. Bronchodilator use in the EMR data was identified by any acute bronchodilator use (via metered dose inhaler or nebulizer) and systemic steroids by medication class of corticosteroids. Nebulization codes used in the claims data were “aerosol or vapor inhalations” and “airway inhalation treatment.” All the above variables were coded as binary variables. Vital signs used for analysis were the initial readings at the time of the ED visit.
Statistical Analysis
Separate analyses were done for EMR and claims data elements. The EMR data set included the ED visit subset with claims data, but the data elements for analysis were from EMR. We compared the predictor variables for those with asthma exacerbation and those without in the ED visit using t-test for continuous variables and chi-square test for categorical variables. We performed unadjusted and adjusted survey logistic regression analyses to identify predictors of asthma exacerbation in an ED visit. Continuous variables were modeled using both linear and quadratic terms to assess for nonlinearity after centering, and the quadratic term was dropped if there was no quadratic association. Model building was done separately for EMR and claims data, by adding covariates (age, sex, race, season, insurance, respiratory rate, pulse rate, temperature, body mass index [BMI], smoking, principal and secondary diagnoses of asthma, respiratory complaint, bronchodilator use, and corticosteroid use) to the model and looking for conditional significance. The final base model included respiratory complaints listed in chief complaints, principal diagnosis, any short-acting bronchodilator, and steroid (oral or intravenous) use in the ED for EMR data; and principal and secondary diagnoses, nebulization, and steroid (oral or intravenous) use in the ED for claims data.
For analysis using survey methodology, some strata were combined to prevent single sampling unit as recommended.[42] These resulted in 15 strata for EMR data and 11 strata for claims data during analysis ([Supplementary Tables S1] and [S2], available in the online version). Weight for each stratum was calculated based on the principle of inverse probability weighting (weight = 1/proportion of patients in the study population for that particular stratum).[40]
[43]
[44] Extreme weight was predetermined to be truncated to the next highest weight or to a preset weight of 10.[45]
[46] Based on this, one extreme weight for EMR data was truncated to 10.02, and to 10 for claims data for regression analysis. However, untruncated weights were used for prevalence estimates. Finite population correction was used as large proportions were sampled. We calculated Cohen's kappa statistic to evaluate the interrater agreement on the chart review conclusion and for assessing agreement between EMR and claims-based measurement of predictor variables.
We were interested in assessing the performance of currently used algorithm elements, both in isolation and combination. We evaluated the performance of 5 models in EMR and 4 models in claims. Model performance discrimination was evaluated using the area under the curve (AUC), which was calculated by nonparametric trapezoidal approximation to the estimated false-positive and true-positive rate points and compared with each other to choose the final predictive model.[47] AUC interpretation for the usefulness of the classifier was considered with the following cut-offs: > 0.9: high accuracy; 0.7 to 0.9: moderate accuracy; and 0.5 to 0.7: low accuracy.[48]
[49] Nomograms were constructed for the final predictive models to provide visual approach for calculating probabilities of ED exacerbation.[50] We compared EMR models to the claims models for the sample with overlapping data. Models were redone for EMR algorithm using this restricted sample for the sake of comparability and AUC was compared.
We performed sensitivity analyses by using different definitions of the predictor variables. For EMR, we utilized only nebulizer administration in the ED instead of any short-acting bronchodilator. For claims, we utilized additional claims submitted on the same day as ED visit claims, which provided separate principal and secondary diagnoses and medications. We combined both the same-day claim files and ED visit claim files and performed the analysis. A p-value of ≤ 0.05 was considered statistically significant. All analyses were done in the Stata statistical software, version 14.2 (StataCorp LP, Texas, United States).
Results
Validation of Chart Review
An independent reviewer evaluated 66 charts previously reviewed by the initial reviewers (29 for one and 37 for another). Agreement between reviewers was 95.45% (63/66) and the kappa statistic was 0.91. The discrepancies were in missing cardiomyopathy which was an exclusion criterion by one; missing asthma exacerbation in the setting of a fall; and allergic reaction to nuts considered as asthma exacerbation, by another.
Characteristics of the Study Sample
There were 966 eligible ED visits in the total population and 731 in the GHP population ([Fig. 1]) after those excluded by chart review (no ED notes, n = 17; COPD, n = 11; coronary artery disease, n = 1; heart failure, n = 4; hospitalization, n = 1). The prevalence of asthma exacerbation in the source population meeting the inclusion criteria, calculated using the EMR data, was 10.84% (95% confidence interval: 10.47–11.22). Mean age of the EMR sample was 22 years, mostly white (93%), never smoked (56%), and female (64%) ([Table 1]). There were seasonal variations in the ED visits studied, with the most common season being the fall, followed by spring. Overall claims sample was similar to the EMR sample in age, sex, race, season, BMI, and smoking status.
Table 1
Selected demographic variables by asthma exacerbation status for EMR and claims data
|
Characteristics
|
Total (n = 966)
|
Asthma exacerbation (n = 497)
|
No asthma exacerbation (n = 469)
|
p-Value[a]
|
|
EMR data
|
|
Age at ED visit (mean, SD in y)
|
22.04 (10.56)
|
21.08 (10.97)
|
23.05 (10.03)
|
0.004
|
|
Age at ED visit (n, column %)
|
|
≤ 18 y
|
374 (38.72)
|
211 (42.45)
|
163 (34.75)
|
0.014
|
|
> 18 y
|
592 (61.28)
|
286 (57.55)
|
306 (65.25)
|
|
Male (n, %)
|
351 (36.34)
|
195 (39.24)
|
156 (33.26)
|
0.05
|
|
White (n, %)
|
898 (92.96)
|
461 (92.76)
|
437 (93.18)
|
0.80
|
|
Season (n, column %)
|
|
Summer
|
203 (21.01)
|
82 (16.50)
|
121 (25.80)
|
0.003
|
|
Fall
|
334 (34.58)
|
176 (35.41)
|
158 (33.69)
|
|
Winter
|
172 (17.81)
|
101 (20.32)
|
71 (15.14)
|
|
Spring
|
257 (26.60)
|
138 (27.77)
|
119 (25.37)
|
|
Has GHP insurance at ED visit (n, %)
|
751 (77.74)
|
371 (74.65)
|
380 (81.02)
|
0.02
|
|
Asthma principal diagnosis (n, %)
|
296 (30.64)
|
267 (53.72)
|
29 (6.18)
|
< 0.001
|
|
Asthma secondary diagnosis (n, %)
|
326 (33.75)
|
116 (23.34)
|
210 (44.78)
|
< 0.001
|
|
Respiratory complaint (n, %)
|
803 (83.13)
|
490 (98.59)
|
313 (66.74)
|
< 0.001
|
|
Bronchodilator use (n, %)
|
459 (47.52)
|
366 (73.64)
|
93 (19.83)
|
< 0.001
|
|
Corticosteroids (n, %)
|
298 (30.85)
|
262 (52.72)
|
36 (7.68)
|
< 0.001
|
|
Bronchodilator use or corticosteroids (n, %)
|
560 (57.97)
|
440 (88.53)
|
120 (25.59)
|
< 0.001
|
|
Initial respiratory rate at ED visit[b] (mean, SD, /min)
|
19.13 (3.60)
|
20.05 (3.82)
|
18.14 (3.06)
|
< 0.001
|
|
Initial pulse rate at ED visit[c] (mean, SD, /min)
|
97.99 (20.69)
|
102.33 (21.07)
|
93.30 (19.21)
|
< 0.001
|
|
Initial temperature at ED visit[d] (mean, /min)
|
98.38 (0.99)
|
98.46 (1.07)
|
98.31 (0.89)
|
0.02
|
|
BMI[e] (kg/m2), mean (SD)
|
28.28 (10.05)
|
28.23 (10.08)
|
28.33 (10.02)
|
0.88
|
|
Smoking[f] (n, column %)
|
|
Never smoker
|
520 (55.97)
|
291 (60.75)
|
229 (50.89)
|
0.009
|
|
Past smoker
|
150 (16.15)
|
67 (13.99)
|
83 (18.44)
|
|
Current smoker
|
259 (27.88)
|
121 (25.26)
|
138 (30.67)
|
|
Claims data
|
Total (
n
= 731)
|
Asthma exacerbation (
n
= 367)
|
No asthma exacerbation (
n
= 364)
|
p
-Value[a]
|
|
Age at ED visit (mean, SD in y)
|
22.24 (10.46)
|
20.85 (10.77)
|
23.65 (9.96)
|
< 0.001
|
|
Age at ED visit (n, column %)
|
|
≤ 18 y
|
283 (38.71)
|
162 (44.14)
|
121 (33.24)
|
0.002
|
|
> 18 y
|
448 (61.29)
|
205 (55.86)
|
243 (66.76)
|
|
Male (n, %)
|
260 (35.57)
|
138 (37.60)
|
122 (33.52)
|
0.25
|
|
White (n, %)
|
682 (93.30)
|
343 (93.46)
|
339 (93.13)
|
0.86
|
|
Season (n, column %)
|
|
Summer
|
161 (22.02)
|
61 (16.62)
|
100 (27.47)
|
0.002
|
|
Fall
|
246 (33.65)
|
123 (33.51)
|
123 (33.79)
|
|
Winter
|
116 (15.87)
|
64 (17.44)
|
52 (14.29)
|
|
Spring
|
208 (28.45)
|
119 (32.43)
|
89 (24.45)
|
|
Asthma principal diagnosis (n, %)
|
229 (31.33)
|
226 (61.58)
|
3 (0.82)
|
< 0.001
|
|
Asthma secondary diagnosis (n, %)
|
108 (14.77)
|
62 (16.89)
|
46 (12.64)
|
0.11
|
|
Nebulization (n, %)
|
315 (43.09)
|
273 (74.39)
|
42 (11.54)
|
< 0.001
|
|
Corticosteroids (n, %)
|
213 (29.14)
|
185 (50.41)
|
28 (7.69)
|
< 0.001
|
|
Nebulization or steroids (n, %)
|
372 (50.89)
|
310 (84.47)
|
62 (17.03)
|
< 0.001
|
|
BMI[g] (kg/m2), mean (SD)
|
28.27 (9.75)
|
28.12 (9.88)
|
28.42 (9.64)
|
0.69
|
|
Smoking[h] (n, column %)
|
|
Never smoker
|
411 (57.89)
|
229 (63.97)
|
182 (51.70)
|
0.003
|
|
Past smoker
|
116 (16.34)
|
54 (15.08)
|
62 (17.61)
|
|
Current smoker
|
183 (25.77)
|
75 (20.95)
|
108 (30.68)
|
Abbreviations: BMI, body mass index; ED, emergency department; EMR, electronic medical record; GHP, Geisinger Health Plan; SD, standard deviation.
a
t-Test for continuous variables and chi-square test for categorical variables comparing asthma exacerbation and no asthma exacerbation.
b Sixteen missing values.
c Twenty-two missing values.
d Fourteen missing values.
e Ninety-eight missing values.
f Thirty-seven missing values.
g Seventy-one missing values.
h Twenty-one missing values.
Fig. 1 Study flowchart.
The asthma exacerbation group in the EMR sample compared with the no asthma exacerbation group was younger, less likely to have the ED visit in the summer, and more likely to be a never smoker. The distribution of those with asthma exacerbation was slightly different for EMR and claims sample, with those in claims being younger, female, with more spring visits, and less current smokers compared with EMR. Asthma principal diagnosis was more likely in the asthma exacerbation EMR sample while secondary diagnosis was more likely in the no asthma exacerbation EMR sample. While 6% of EMR sample with asthma principal diagnosis did not have asthma exacerbation, less than 1% of claims sample with principal diagnosis did not have asthma exacerbation. In the EMR sample, most of those with asthma exacerbation had a respiratory complaint (99%), while three-quarters of them (74%) had bronchodilator use and half of them had corticosteroid use (53%) and principal diagnosis (54%). The initial respiratory and pulse rates and temperature were higher in the asthma exacerbation group (p < 0.05 for all). In the claims sample, three-quarters of the asthma exacerbation group had nebulization, while two-thirds had principal diagnosis and half of them had corticosteroid use.
Concordance between EMR and Claims Data
We evaluated the concordance in documentation between EMR and claims data among the overlapping 731 ED visits ([Supplementary Table S3], available in the online version). There was 96% concordance for principal diagnosis (perfect agreement using κ statistic); 38% (n = 10) of those with principal diagnosis present in EMR but missed by claims had asthma exacerbation while 100% (n = 6) for vice versa. There was 77% concordance for secondary diagnosis (moderate agreement); 30% (n = 50) of those with secondary diagnosis in EMR and missed by claims had asthma exacerbation while 100% (n = 4) for vice versa. There was 84% concordance for nebulization (substantial agreement); 74% (n = 50) of those with nebulization use in EMR but missed by claims had asthma exacerbation while 92% (n = 44) for vice versa. There was 91% concordance for corticosteroids (substantial agreement); 87% (n = 39) of those with steroids use in EMR but missed by claims had asthma exacerbation while 75% (n = 15) for vice versa.
Unadjusted and Adjusted Associations of Predictors
All unadjusted predictors evaluated that showed an association with asthma exacerbation-related ED visit in both claims and EMR data are shown in [Table 2], with most of them showing a positive association. In the adjusted analysis, principal diagnosis had the highest odds ratio (OR) among the different predictors in EMR and claims data, followed by bronchodilator use. Among the vital signs, the odds of asthma exacerbation-related ED visit increased by 12% for every breath/minute increase in the initial respiratory rate (OR = 1.12 [1.05, 1.19]).
Table 2
Unadjusted and adjusted associations[a] by survey logistic regression[b] of selected predictor variables with asthma exacerbation-related ED visit
|
Characteristics–EMR data
|
Unadjusted association OR (95% CI)
|
Adjusted association OR (95% CI)
|
|
Asthma principal diagnosis
|
46.19 (31.61, 67.50)
|
29.16 (14.97, 56.81)
|
|
Asthma secondary diagnosis
|
0.70 (0.58, 0.83)
|
0.85 (0.53, 1.38)
|
|
Respiratory complaint
|
68.37 (34.92, 133.87)
|
6.09 (2.30, 16.14)
|
|
Bronchodilator use
|
41.05 (30.73, 54.84)
|
23.67 (12.99, 43.11)
|
|
Corticosteroids
|
21.24 (13.92, 32.40)
|
6.06 (3.34, 11.01)
|
|
Initial respiratory rate
|
1.29 (1.22, 1.37)
|
1.12 (1.05, 1.19)
|
|
Initial pulse rate
|
1.03 (1.02, 1.04)
|
1.01 (1.00, 1.02)
|
|
Initial temperature
|
1.22 (1.05, 1.41)
|
1.24 (0.98, 1.57)
|
|
Smoking
|
|
Never smoker
|
Reference
|
Reference
|
|
Past smoker
|
0.66 (0.44, 0.98)
|
0.57 (0.26, 1.23)
|
|
Current smoker
|
0.56 (0.41, 0.77)
|
0.66 (0.41, 1.05)
|
|
Characteristics–Claims data
|
Unadjusted association OR (95% CI)
|
Adjusted association OR (95% CI)
|
|
Asthma principal diagnosis
|
401.88 (228.34, 707.31)
|
305.56 (158.11, 590.53)
|
|
Asthma secondary diagnosis
|
1.86 (1.47, 2.35)
|
6.63 (4.53, 9.70)
|
|
Nebulization
|
45.92 (38.55, 54.69)
|
32.88 (25.39, 42.58)
|
|
Corticosteroids
|
19.75 (15.16, 25.73)
|
4.39 (2.98, 6.46)
|
Abbreviations: CI, confidence interval; ED, emergency department; EMR, electronic medical record; OR, odds ratio.
a Adjusted associations are for each variable in the final model or added to the final model. Final model variables were respiratory complaints (like cough, congestion, wheezing, short of breath, etc.), principal diagnosis, any short-acting bronchodilator, and steroid (oral or intravenous) use in the ED for EMR data; and principal and secondary diagnoses, nebulization, and steroid (oral or intravenous) use in the ED for claims data.
b Survey logistic regression was used for the analysis; weighted based on sampling proportions; accounted for stratified survey sampling.
Receiver Operating Characteristic Curves
EMR Data
We evaluated the AUC for five receiver operating characteristic (ROC) curves with different combinations of commonly used predictors of asthma exacerbation-related ED visit in the EMR, and the final base model chosen was respiratory complaint, primary diagnosis, any short-acting bronchodilator use, and oral or intravenous steroids ([Fig. 2]). The model with initial respiratory rate in addition to the base model had statistically significantly better performance than the base model (p = 0.04) with ROC for the former being 0.93 (0.92, 0.95) versus 0.93 (0.91, 0.94) for the latter. At a probability cut-off of 0.55 for the former model, sensitivity was 83.98%, specificity was 82.93%, PPV was 84.15%, and NPV was 82.75%. However, given that the gains in AUC were minimal when compared with the difficulty of obtaining initial vital sign for routine implementation purposes when an algorithm needs to be used in a large scale, we decided to use the simpler latter model as the final model. Different probability cut-offs were evaluated for the final model to optimize the sensitivity and specificity ([Table 3]). Probability cut-off of 0.5 had the highest sensitivity of 95.57% and 0.7 had the highest PPV of 94.15%. The regression equation for the final model was: probability of asthma exacerbation =
Table 3
Sensitivity, specificity, positive predictive value, and negative predictive value for the final models in EMR for different asthma exacerbation probability cut-offs
|
Classification function
|
Probability cut-offs for EMR data final model
|
|
0.5
|
0.6
|
0.7
|
|
Sensitivity
|
95.57%
|
70.62%
|
61.57%
|
|
Specificity
|
73.56%
|
90.62%
|
95.95%
|
|
Positive predictive value
|
79.30%
|
88.86%
|
94.15%
|
|
Negative predictive value
|
94.01%
|
74.43%
|
70.20%
|
Abbreviation: EMR, electronic medical record.
Fig. 2 Receiver operating characteristic (ROC) curves for electronic medical record data.
A nomogram was constructed to calculate the predicted probability of asthma exacerbation-related ED visit ([Fig. 3]). To interpret the nomogram, first the score for each variable in the model should be calculated by lining up the value for the variable with the score scale in the upper part of the nomogram, and then all the scores should be summed. The resulting total score should be matched to the probability of asthma exacerbation in the lower part of the nomogram.[51] Binary probability cut-off to indicate asthma exacerbation can be chosen for each study based on the desired test statistics of sensitivity, specificity, and predictive values as in [Table 3]. Each patient's probability of asthma exacerbation obtained from the nomogram can finally be converted to “yes or no” for asthma exacerbation based on the predetermined probability cut-off.
Fig. 3 Nomogram for electronic medical record data.
Claims Data
Multiple models conventionally used in studies were evaluated for claims data also, and the final base model chosen was principal and secondary diagnoses, bronchodilator, and oral or intravenous steroid use ([Fig. 4]). An extended model with centered age, squared centered age, race, and season as nonordinal variable in addition to the base model was the best model (p = 0.046), with ROC for final model being 0.94 (0.93, 0.96) versus 0.95 (0.93, 0.96) for the extended model. Again, for the sake of simplicity for large-scale implementation, we did not choose the extended model for the final model. Different probability cut-offs were evaluated for the final model, with 0.5 being the most sensitive model at 91% and 0.6 having the highest PPV at 95% ([Table 4]). Nomogram was constructed to calculate the predicted probability of asthma exacerbation-related ED visit ([Fig. 5]).
Fig. 4 Receiver operating characteristic (ROC) curves for claims data.
Fig. 5 Nomogram for claims data.
Table 4
Sensitivity, specificity, positive predictive value, and negative predictive value for the final models in claims for different asthma exacerbation probability cut-offs
|
Classification function
|
Probability cut-offs for claims data final model
|
|
0.5
|
0.55
|
0.6
|
|
Sensitivity
|
91.28%
|
88.83%
|
73.57%
|
|
Specificity
|
87.36%
|
88.19%
|
96.15%
|
|
Positive predictive value
|
87.93%
|
88.35%
|
95.07%
|
|
Negative predictive value
|
90.86%
|
88.67%
|
78.30%
|
The regression equation for the final model was: probability of asthma exacerbation =
Sensitivity Analysis
Sensitivity analysis did not change the algorithm performance. Using nebulizer administration instead of any short-acting bronchodilator use for EMR data, reduced the performance of the EMR model nonsignificantly with AUC of 0.92 versus 0.93 (p = 0.14). Using same-day claim files along with ED visit claims did not improve the performance of the claims algorithm—AUC of 0.94 for both algorithms, p = 0.91 for the difference.
Comparison of EMR and Claims Data Algorithms
We compared the final models from EMR and claims data and there was no statistically significant difference, p = 0.54 ([Fig. 6]). AUC for the model with primary diagnosis only was better for claims data (0.80, 95% confidence interval [0.78, 0.83]) compared with EMR data (0.79 [0.76, 0.81]), p = 0.03. AUC for the model with primary and secondary diagnoses only was also better for claims data (0.84 [0.81, 0.87]) compared with EMR data (0.77 [0.74, 0.80]), p ≤ 0.001. There was no statistically significant difference in AUC for the model with bronchodilator and steroid only based on EMR (0.88 [0.85, 0.90]) and claims (0.87 [0.84, 0.89]), p = 0.45. Similarly, no difference was found for model with bronchodilator only. However, AUC for the model with steroid only was better for EMR (0.74 [0.72, 0.77]) than claims (0.71 [0.68, 0.74]), p = 0.004.
Fig. 6 Comparison of receiver operating characteristic (ROC) curves of the final model for electronic medical record (EMR) and claims data.
Discussion
We present the largest study to our knowledge validating algorithms to identify asthma exacerbation-related ED visit in pediatric and adult population with separate algorithms for EMR and claims data. These algorithms use readily available inputs like chief complaints, diagnosis codes, and medication use. Depending on the proposed utility of the algorithm, different probability cut-offs can be set. While population-based epidemiology studies and quality improvement projects require high sensitivity, performance metrics for health system and genetic epidemiology studies require algorithms with high PPV and specificity. We have provided nomograms which can be utilized with different probability cut-offs for these different purposes. As examples, genetic association study of asthma exacerbation or a performance metric for asthma care will need a high PPV of at least 90%[52] and the possible probability cut-offs for the algorithms would be 0.7 for EMR and 0.6 for claims data to identify asthma exacerbation; however, for a quality improvement project involving escalation of care for asthmatics with an asthma exacerbation leading to ED visit, high sensitivity would be the priority so that most of the eligible patients would receive the needed care; in this situation, probability cut-off of 0.5 might be more appropriate for EMR and claims data. The algorithm could also be integrated into a clinical decision support with a probability cut-off of 0.5, for managing asthma patients with respiratory complaints in an outpatient setting to initiate aggressive management, as a patient with a previous history of asthma exacerbation is at high risk of another exacerbation requiring ED visit.
The only previous algorithms to identify asthma exacerbation leading to ED visit were both performed in pediatric population[31]
[33] and ours involves both adult and pediatric age ranges. Regarding the performance of our algorithm, as it pertains to generalizability, sensitivity and specificity are fixed test characteristics. However, the predictive values are dependent on the prevalence of asthma exacerbation, which is variable by the region of the United States, ranging from 43.2% in the Northeast, 44.7% Midwest to 48.8% in the South and West.[53] Our study is the first in the Northeast, and has better predictive value for the same sensitivity (94% vs. 79%) even though our asthma attack prevalence is the lowest of the regions of the United States.[31] Our health system also encounters predominantly rural[54] and white population,[9] which has been shown in previous studies to have a lower prevalence of asthma exacerbation and, therefore, could lower the PPV of our algorithm. Despite the availability of effective preventive therapy and guidelines, asthma management and control in the United States is unsatisfactory and costs associated with asthma are increasing.[55]
[56] It has been shown that decision support tools, feedback and audit, and clinical pharmacy support were crucial to encourage adherence to asthma guidelines.[57] Our algorithm with acceptable PPV helps to identify the high-risk patients, making these tools possible, and has the potential to improve asthma management.
EMR and claims data each have their advantages and disadvantages, while the former is able to provide better clinical data like vital signs in asthma exacerbation, the latter is able to provide a complete picture of a patient's care. The concordance between EMR- and claims-based variables for asthma exacerbation identification was variable, with > 90% concordance for primary diagnosis and corticosteroids. When we analyzed how predictive a characteristic that was present in one of the data sets (EMR or claims) but missed by the other was, claim-positive, EMR-negative subset had higher rates of asthma exacerbation for principal diagnosis, secondary diagnosis, and bronchodilator, while EMR-positive, claims-negative subset, had for steroids. Secondary asthma diagnosis was not predictive in the EMR data but predictive in the claims data. Previous studies have compared EMR to claims data and have found variable concordance based on medication or disease condition.[58]
[59] Models that use diagnostic codes only to predict asthma exacerbation performed better in claims data than EMR, while steroid-only model performed better in EMR data, albeit the AUC of such models were inferior to our final model. This shows the strengths and weaknesses of data elements from EMR and claims data. The final model from EMR and claims were not statistically different. The AUC for the final model for EMR improved from 0.93 (0.91, 0.94) to 0.95 (0.93, 0.96) when the sample used to derive the models changed to a restricted sample with overlapping data. Even though the AUC changed, the change is within acceptable limits of random variability as assessed by the 95% confidence intervals which showed overlapping intervals.
Strengths of our study include separate algorithms for EMR and claims data, validation against a large gold standard population, inclusion of both pediatric and adult population, using appropriate statistical methods to refer to the source population to calculate all performance metrics, and creation of nomograms for use by others. Our study is not without limitations in that the diagnostic codes we used for the algorithm were ICD-9 codes, which at the current time makes our algorithm relevant only for retrospective work, if used as studied. Given the current standard of using ICD-10 codes, the asthma code ICD-10 code J45.xx should be used instead of ICD-9 code 493.xx when using the algorithm, to make it relevant for prospective work. Also, our gold standard did not use peak flow data to confirm asthma exacerbation but we used extensive clinical data to classify our gold standard and validated this.[34] We also oversampled ED visits with classifiers of interest, but we were able to calculate estimates of classification function in the source population, making our predictive values generalizable to an ED asthma population similar to ours, which is predominantly rural and white. We also did not explore data mining and analytics-based techniques for classification such as decision trees, Support Vector Machines, or newer advanced techniques that may provide better performance, and these could be evaluated in future studies. The agreement between EMR and claims data for various data elements could have been influenced by (1) our EMR system, as billing is highly integrated with EMR for Epic Systems Corporation-based software, and (2) by our organization structure which is an integrated health system providing both health care and insurance, influencing generalizability as it pertains to comparison of EMR and claims data elements.
Conclusion
We were able to create an algorithm based on respiratory complaints listed in chief complaints, principal diagnosis, any short-acting bronchodilator, and steroid (oral or intravenous) use in the ED for EMR data, with an excellent PPV of 94%; and another based on principal and secondary diagnoses, nebulization, and steroid (oral or intravenous) use in the ED for claims data, with an excellent PPV of 95%. We also provide different probability cut-offs to provide a sensitive or specific algorithm based on the needs of a research study or quality improvement program.
Clinical Relevance Statement
Clinical Relevance Statement
Asthma exacerbation leading to ED visit is prevalent and preventable. Identification of these asthma exacerbations using EMR and claims data has the potential to improve morbidity in asthma by integrated care and better research. This study provides separate algorithms to identify asthma-related ED visit using EMR and claims data with high positive predictive value using readily available data elements like chief complaints, diagnostic codes, and medications.
Multiple Choice Question
What was the purpose of nomogram in this study?
-
To provide a mathematical formula for calculating predicted probability of asthma exacerbation in the ED.
-
To provide a visual approach for calculating predicted probability of asthma exacerbation in the ED.
-
To provide a mathematical formula for calculating area under the curve for the final model to predict asthma exacerbation in the ED.
-
To provide a visual approach for calculating area under the curve for the final model to predict asthma exacerbation in the ED.
Correct Answer: The correct answer is option b, to provide a visual approach for calculating predicted probability of asthma exacerbation in the ED. Nomograms were constructed in this study to provide a visual approach for calculating predicted probabilities of asthma exacerbation-related ED visit, although traditionally they have used in medicine to estimate prognosis. The advantage of nomogram is the ability to calculate probabilities based on individual patient's characteristics of different predictor variables. A nomogram can also be integrated into clinical decision support tools and utilized for clinical care. To interpret the nomogram, first the score for each variable in the model should be calculated by lining up the value for the variable with the score scale in the upper part of the nomogram, and then all the scores should be summed. The resulting total score should be matched to the probability of asthma exacerbation in the lower part of the nomogram. Binary probability cut-off to indicate asthma exacerbation can be chosen for each project based on the desired test statistics of sensitivity, specificity, and predictive values. Each patient's probability of asthma exacerbation obtained from the nomogram can finally be converted to “yes or no” for asthma exacerbation based on the predetermined probability cut-off.
