The accuracy of an electronic Pulmonary Embolism Severity Index auto-populated from the electronic health record

D.R. Vinson; J.E. Morley; J. Huang; V. Liu; M.L. Anderson; C. E. Drenten; R.P. Radecki; D.K. Nishijima; M.E. Reed; the Kaisers Permanente CREST Network; on behalf of

doi:10.4338/ACI-2014-12-RA-0116

Applied Clinical Informatics, Table of Contents

Appl Clin Inform 2015; 06(02): 318-333
DOI: 10.4338/ACI-2014-12-RA-0116

Research Article

Schattauer GmbH

The accuracy of an electronic Pulmonary Embolism Severity Index auto-populated from the electronic health record

Setting the stage for computerized clinical decision support

D.R. Vinson

¹The Permanente Medical Group, Oakland, California

²Department of Emergency Medicine, Kaiser Permanente Roseville Medical Center, Roseville, California

³Kaiser Permanente Division of Research, Oakland, California

,

J.E. Morley

⁴Department of Emergency Medicine, University of California Davis School of Medicine, Sacramento, California

,

J. Huang

³Kaiser Permanente Division of Research, Oakland, California

,

V. Liu

¹The Permanente Medical Group, Oakland, California

³Kaiser Permanente Division of Research, Oakland, California

⁵Department of Pulmonary and Critical Care Medicine, Kaiser Permanente Santa Clara Medical Center, Santa Clara, California

,

M.L. Anderson

¹The Permanente Medical Group, Oakland, California

²Department of Emergency Medicine, Kaiser Permanente Roseville Medical Center, Roseville, California

,

C. E. Drenten

⁶Department of Emergency Medicine, Sutter General Medical Center, Sacramento, California

,

R.P. Radecki

⁷Department of Emergency Medicine, The University of Texas Medical School, Houston, Texas

,

D.K. Nishijima

⁴Department of Emergency Medicine, University of California Davis School of Medicine, Sacramento, California

,

M.E. Reed

³Kaiser Permanente Division of Research, Oakland, California

,

the Kaisers Permanente CREST Network

› Author Affiliations

Abstract

Summary

Background: The Pulmonary Embolism (PE) Severity Index identifies emergency department (ED) patients with acute PE that can be safely managed without hospitalization. However, the Index comprises 11 weighted variables, complexity that can impede its integration into contextual work-flow.

Objective: We designed a computerized version of the PE Severity Index (e-Index) to automatically extract the required variables from discrete fields in the electronic health record (EHR). We tested the e-Index on the study population to determine its accuracy compared with a gold standard generated by physician abstraction of the EHR on manual chart review.

Methods: This retrospective cohort study included adults with objectively-confirmed acute PE in four community EDs from 2010–2012. Outcomes included performance characteristics of the e-Index for individual values, the number of cases requiring physician editing, and the accuracy of the e-Index risk category (low vs. higher).

Results: For the 593 eligible patients, there were 6,523 values automatically extracted. Fifty one of these needed physician editing, yielding an accuracy at the value-level of 99.2% (95% confidence interval [CI], 99.0%-99.4%). Sensitivity was 96.9% (95% CI, 96.0%-97.9%) and specificity was 99.8% (95% CI, 99.7%-99.9%). The 51 corrected values were distributed among 47 cases: 43 cases required the correction of one variable and four cases required the correction of two. At the risk-category level, the e-Index had an accuracy of 96.8% (95% CI, 95.0%-98.0%), under-classifying 16 higher-risk cases (2.7%) and over-classifying 3 low-risk cases (0.5%).

Conclusion: Our automated extraction of variables from the EHR for the e-Index demonstrates substantial accuracy, requiring a minimum of physician editing. This should increase user acceptability and implementation success of a computerized clinical decision support system built around the e-Index, and may serve as a model to automate other complex risk stratification instruments.

Citation: Vinson DR, Morley JE, Huang J, Liu V, Anderson ML, Drenten CE, Radecki RP, Nishijima DK, Reed ME. The accuracy of an electronic pulmonary embolism severity index auto-populated from the electronic health record. Appl Clin Inf 2015; 6: 318–333

http://dx.doi.org/10.4338/ACI-2014-12-RA-0116

Keywords

Clinical decision support systems - electronic health record - risk assessment - pulmonary embolism - emergency medicine - data completeness

Full Text

References

References
1 Aujesky D, Roy PM, Verschuren F, Righini M, Osterwalder J, Egloff M, Renaud B, Verhamme P, Stone RA, Legall C, Sanchez O, Pugh NA, N’Gako A, Cornuz J, Hugli O, Beer HJ, Perrier A, Fine MJ, Yealy DM. Out-patient versus inpatient treatment for patients with acute pulmonary embolism: an international, open-label, randomised, non-inferiority trial. Lancet 2011; 378: 41-48.
2 Vinson DR, Zehtabchi S, Yealy DM. Can selected patients with newly diagnosed pulmonary embolism be safely treated without hospitalization? A systematic review. Ann Emerg Med 2012; 60: 651-662.
3 Moja L, Liberati EG, Galuppo L, Gorli M, Maraldi M, Nanni O, Rigon G, Ruggieri P, Ruggiero F, Scaratti G, Vaona A, Kwag KH. Barriers and facilitators to the uptake of computerized clinical decision support systems in specialty hospitals: protocol for a qualitative cross-sectional study. Implement Sci 2014; 9: 105.
4 Navar-Boggan AM, Rymer JA, Piccini JP, Shatila W, Ring L, Stafford JA, Al-Khatib SM, Peterson ED. Accuracy and validation of an automated electronic algorithm to identify patients with atrial fibrillation at risk for stroke. Am Heart J 2015; 169: 39-44.
5 Thigpen JL, Dillon C, Forster KB, Henault L, Quinn EK, Tripodis Y, Berger PB, Hylek EM, Limdi NA. Validity of International Classification of Disease Codes to Identify Ischemic Stroke and Intracranial Hemorrhage Among Individuals With Associated Diagnosis of Atrial Fibrillation. Circ Cardiovasc Qual Outcomes 2015; 8: 8-14.
6 McCormick N, Lacaille D, Bhole V, Avina-Zubieta JA. Validity of heart failure diagnoses in administrative databases: a systematic review and meta-analysis. PLoS One 2014; 9: e104519.
7 Nouraei SA, Hudovsky A, Frampton AE, Mufti U, White NB, Wathen CG, Sandhu GS, Darzi A. A Study of Clinical Coding Accuracy in Surgery: Implications for the Use of Administrative Big Data for Outcomes Management. Ann Surg 2014 Dec 2 [Epub ahead of print].
8 Weiskopf NG, Hripcsak G, Swaminathan S, Weng C. Defining and measuring completeness of electronic health records for secondary use. J Biomed Inform 2013; 46: 830-836.
9 Gershon AS, Wang C, Guan J, Vasilevska-Ristovska J, Cicutto L, To T. Identifying individuals with physcian diagnosed COPD in health administrative databases. COPD 2009; 6: 388-394.
10 Gershon AS, Wang C, Guan J, Vasilevska-Ristovska J, Cicutto L, To T. Identifying patients with physician-diagnosed asthma in health administrative databases. Can Respir J 2009; 16: 183-188.
11 Wright A, Pang J, Feblowitz JC, Maloney FL, Wilcox AR, McLoughlin KS, Ramelson H, Schneider L, Bates DW. Improving completeness of electronic problem lists through clinical decision support: a randomized, controlled trial. J Am Med Inform Assoc 2012; 19: 555-561.
12 Bakel LA, Wilson K, Tyler A, Tham E, Reese J, Bothner J, Kaplan DW. A quality improvement study to improve inpatient problem list use. Hosp Pediatr 2014; 4: 205-210.
13 Bornstein S. An integrated EHR at Northern California Kaiser Permanente: Pitfalls, challenges, and benefits experienced in transitioning. Appl Clin Inform 2012; 3: 318-325.
14 Vinson DR, Drenten CE, Huang J, Morley JE, Anderson ML, Reed ME, Nishijima DK, Liu V. on behalf of the Kaisers Permanente CREST Network. Impact of Relative Contraindications to Home Management in Emergency Department Patients with Low-Risk Pulmonary Embolism. Ann Am Thorac Soc. 2015 Feb 19 [epub ahead of print].
15 Aujesky D, Obrosky DS, Stone RA, Auble TE, Perrier A, Cornuz J, Roy PM, Fine MJ. Derivation and validation of a prognostic model for pulmonary embolism. Am J Respir Crit Care Med 2005; 172: 1041-1046.
16 Jimenez D, Aujesky D, Moores L, Gomez V, Lobo JL, Uresandi F, Otero R, Monreal M, Muriel A, Yusen RD. Simplification of the pulmonary embolism severity index for prognostication in patients with acute symptomatic pulmonary embolism. Arch Intern Med 2010; 170: 1383-1389.
17 Briese B, Schreiber D, Lin B, Liu G, Fansler J, Goldhaber S, O’Neil B, Slattery D, Hiestand B, Kline J, Pol-lack C.. Derivation of a simplified Pulmonary Embolism Triage Score (PETS) to predict the mortality in patients with confirmed pulmonary embolism from the Emergency Medicine Pulmonary Embolism in the Real World Registry (EMPEROR). Acad Emerg Med 2012; 19: S143-S144.
18 Lobach D, Sanders GD, Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux R, Samsa G, Hasselblad V, Williams JW, Wing L, Musty M, Kendrick AS. Enabling health care decisionmaking through clinical decision support and knowledge management. Evid Rep Technol Assess (Full Rep) 2012: 1-784.
19 Bishop RO, Patrick J, Besiso A. Efficiency achievements from a user-developed real-time modifiable clinical information system. Ann Emerg Med 2015; 65: 133-142.
20 Horsky J, Phansalkar S, Desai A, Bell D, Middleton B. Design of decision support interventions for medication prescribing. Int J Med Inform 2013; 82: 492-503.
21 Jones BE, Jones J, Bewick T, Lim WS, Aronsky D, Brown SM, Boersma WG, van der Eerden MM, Dean NC. CURB-65 pneumonia severity assessment adapted for electronic decision support. Chest 2011; 140: 156-163.
22 Fine MJ, Auble TE, Yealy DM, Hanusa BH, Weissfeld LA, Singer DE, Coley CM, Marrie TJ, Kapoor WN. A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med 1997; 336: 243-250.
23 Lim WS, van der Eerden MM, Laing R, Boersma WG, Karalus N, Town GI, Lewis SA, Macfarlane JT. Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study. Thorax 2003; 58: 377-382.
24 Olesen JB, Lip GY, Hansen ML, Hansen PR, Tolstrup JS, Lindhardsen J, Selmer C, Ahlehoff O, Olsen AM, Gislason GH, Torp-Pedersen C. Validation of risk stratification schemes for predicting stroke and thromboembolism in patients with atrial fibrillation: nationwide cohort study. BMJ 2011; 342: d124.
25 Singer DE, Chang Y, Borowsky LH, Fang MC, Pomernacki NK, Udaltsova N, Reynolds K, Go AS. A new risk scheme to predict ischemic stroke and other thromboembolism in atrial fibrillation: the ATRIA study stroke risk score. J Am Heart Assoc 2013; 2: e000250.
26 Pisters R, Lane DA, Nieuwlaat R, de Vos CB, Crijns HJ, Lip GY. A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro Heart Survey. Chest 2010; 138: 1093-1100.
27 Murphy EV. Clinical decision support: effectiveness in improving quality processes and clinical outcomes and factors that may influence success. Yale J Biol Med 2014; 87: 187-197.
28 Moja L, Kwag KH, Lytras T, Bertizzolo L, Brandt L, Pecoraro V, Rigon G, Vaona A, Ruggiero F, Mangia M, Iorio A, Kunnamo I, Bonovas S. Effectiveness of Computerized Decision Support Systems Linked to Electronic Health Records: A Systematic Review and Meta-Analysis. Am J Public Health 2014: e1-e11.
29 Khorasani R, Hentel K, Darer J, Langlotz C, Ip IK, Manaker S, Cardella J, Min R, Seltzer S. Ten commandments for effective clinical decision support for imaging: enabling evidence-based practice to improve quality and reduce waste. AJR Am J Roentgenol 2014; 203: 945-951.
30 Sedlmayr B, Patapovas A, Kirchner M, Sonst A, Muller F, Pfistermeister B, Plank-Kiegele B, Vogler R, Criegee-Rieck M, Prokosch HU, Dormann H, Maas R, Burkle T. Comparative evaluation of different medication safety measures for the emergency department: physicians’ usage and acceptance of training, poster, checklist and computerized decision support. BMC Med Inform Decis Mak 2013; 13: 79.
31 Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, Samsa G, Hasselblad V, Williams JW, Musty MD, Wing L, Kendrick AS, Sanders GD, Lobach D. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157: 29-43.
32 Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, Spurr C, Khorasani R, Tanasijevic M, Middleton B. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10: 523-530.
33 Jia PL, Zhang PF, Li HD, Zhang LH, Chen Y, Zhang MM. Literature review on clinical decision support system reducing medical error. J Evid Based Med 2014; 7: 219-226.
34 Dean NC, Jones BE, Ferraro JP, Vines CG, Haug PJ. Performance and utilization of an emergency department electronic screening tool for pneumonia. JAMA Intern Med 2013; 173: 699-701.
35 Jones BE, Jones JP, Vines CG, Dean NC. Validating hospital admission criteria for decision support in pneumonia. BMC Pulm Med 2014; 14: 149.
36 Dean NC, Jones BE, Jones JP, Ferraro JP, Post HB, Aronsky D, Vines CG, Allen TL, Haug PJ. Impact of an Electronic Clinical Decision Support Tool for Emergency Department Patients With Pneumonia. Ann Emerg Med. 2015 Feb 26 [Epub ahead of print].
37 Mann DM, Kannry JL, Edonyabo D, Li AC, Arciniega J, Stulman J, Romero L, Wisnivesky J, Adler R, McGinn TG. Rationale, design, and implementation protocol of an electronic health record integrated clinical prediction rule (iCPR) randomized trial in primary care. Implement Sci 2011; 6: 109.
38 Handel DA, Wears RL, Nathanson LA, Pines JM. Using information technology to improve the quality and safety of emergency care. Acad Emerg Med 2011; 18: e45-e51.
39 Mishuris RG, Linder JA, Bates DW, Bitton A. Using electronic health record clinical decision support is associated with improved quality of care. Am J Manag Care 2014; 20: e445-e452.