Automated Screening of Emergency Department Notes for Drug-Associated Bleeding Adverse Events Occurring in Older Adults

Abstract

Objective To conduct research to show the value of text mining for automatically identifying suspected bleeding adverse drug events (ADEs) in the emergency department (ED).

Methods A corpus of ED admission notes was manually annotated for bleeding ADEs. The notes were taken for patients ≥ 65 years of age who had an ICD-9 code for bleeding, the presence of hemoglobin value ≤ 8 g/dL, or were transfused > 2 units of packed red blood cells. This training corpus was used to develop bleeding ADE algorithms using Random Forest and Classification and Regression Tree (CART). A completely separate set of notes was annotated and used to test the classification performance of the final models using the area under the ROC curve (AUROC).

Results The best performing CART resulted in an AUROC on the training set of 0.882. The model's AUROC on the test set was 0.827. At a sensitivity of 0.679, the model had a specificity of 0.908 and a positive predictive value (PPV) of 0.814. It had a relatively simple and intuitive structure consisting of 13 decision nodes and 14 leaf nodes. Decision path probabilities ranged from 0.041 to 1.0. The AUROC for the best performing Random Forest method on the training set was 0.917. On the test set, the model's AUROC was 0.859. At a sensitivity of 0.274, the model had a specificity of 0.986 and a PPV of 0.92.

Conclusion Both models accurately identify bleeding ADEs using the presence or absence of certain clinical concepts in ED admission notes for older adult patients. The CART model is particularly noteworthy because it does not require significant technical overhead to implement. Future work should seek to replicate the results on a larger test set pulled from another institution.

Keywords

hemorrhage - quality improvement - text mining - adverse drug event - emergency department

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by the University of Pittsburgh Institutional Review Board.

• References

• 1 US Department of Health and Human Services. National action plan for adverse drug event prevention. 2014 . Available at: https://health.gov/hcq/pdfs/ade-action-plan-508c.pdf
• 2 Dormann H, Sonst A, Müller F. , et al. Adverse drug events in older patients admitted as an emergency: the role of potentially inappropriate medication in elderly people (PRISCUS). Dtsch Arztebl Int 2013; 110 (13) 213-219
• 3 Pretorius RW, Gataric G, Swedlund SK, Miller JR. Reducing the risk of adverse drug events in older adults. Am Fam Physician 2013; 87 (05) 331-336
• 4 Institute for Safe Medication Practices. Safe medication practices high alert medications in acute care settings. 2017 . Available at: http://www.ismp.org/Tools/institutionalhighAlert.asp . Accessed July 28, 2014
• 5 Moriarty JP, Daniels PR, Manning DM. , et al. Going beyond administrative data: retrospective evaluation of an algorithm using the electronic health record to help identify bleeding events among hospitalized medical patients on warfarin. Am J Med Qual 2017; 32 (04) 391-396
• 6 Cullen DJ, Bates DW, Small SD, Cooper JB, Nemeskal AR, Leape LL. The incident reporting system does not detect adverse drug events: a problem for quality improvement. Jt Comm J Qual Improv 1995; 21 (10) 541-548
• 7 Kane-Gill SL, Devlin JW. Adverse drug event reporting in intensive care units: a survey of current practices. Ann Pharmacother 2006; 40 (7-8): 1267-1273
• 8 Jha AK, Kuperman GJ, Teich JM. , et al. Identifying adverse drug events: development of a computer-based monitor and comparison with chart review and stimulated voluntary report. J Am Med Inform Assoc 1998; 5 (03) 305-314
• 9 LePendu P, Iyer SV, Bauer-Mehren A. , et al. Pharmacovigilance using clinical notes. Clin Pharmacol Ther 2013; 93 (06) 547-555
• 10 Warrer P, Hansen EH, Juhl-Jensen L, Aagaard L. Using text-mining techniques in electronic patient records to identify ADRs from medicine use. Br J Clin Pharmacol 2012; 73 (05) 674-684
• 11 Wiley LK, Moretz JD, Denny JC, Peterson JF, Bush WS. Phenotyping Adverse Drug Reactions: Statin-Related Myotoxicity. AMIA Jt Summits Transl Sci Proc 2015; 2015: 466-470
• 12 Kane-Gill SL, MacLasco AM, Saul MI. , et al. Use of text searching for trigger words in medical records to identify adverse drug reactions within an intensive care unit discharge summary. Appl Clin Inform 2016; 7 (03) 660-671
• 13 Rochefort CM, Buckeridge DL, Forster AJ. Accuracy of using automated methods for detecting adverse events from electronic health record data: a research protocol. Implement Sci 2015; 10: 5
• 14 Breiman L, Friedman JH, Olshen RA, Stone CJ. Classification and Regression Trees. Belmont, CA: Wadsworth International Group; 1984
• 15 Tseytlin E, Mitchell K, Legowski E, Corrigan J, Chavan G, Jacobson RS. NOBLE – Flexible concept recognition for large-scale biomedical natural language processing. BMC Bioinformatics. January 14, 2016:17 [cited October 6, 2016]. Available at: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4712516/
• 16 Bodenreider O. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res 2004; 32 (Database issue): D267-D270
• 17 RxNorm Technical Documentation. Available at: http://www.nlm.nih.gov/research/umls/rxnorm/docs/2015/rxnorm_doco_full_2015-1.html . Accessed October 10, 2016
• 18 The International Health Terminology Standards Development Organisation. SNOMED-CT. 2016. Available at: http://www.ihtsdo.org/snomed-ct
• 19 International Conference on Harmonisation. MedDRA [Internet]. Welcome to MedDRA. 2016 [cited October 10, 2016]. Available at: http://www.meddra.org/ . Accessed October 10, 2016
• 20 National Library of Medicine. Medical Subject Headings - Home Page. Welcome to Medical Subject Headings! 2016. Available from: https://www.nlm.nih.gov/mesh/ . Accessed October 10, 2016
• 21 Chapman WW, Bridewell W, Hanbury P, Cooper GF, Buchanan BG. A simple algorithm for identifying negated findings and diseases in discharge summaries. J Biomed Inform 2001; 34 (05) 301-310
• 22 Hall MA. Correlation-based Feature Subset Selection for Machine Learning. Hamilton, NZ: University of Waikato; 1998
• 23 WEKA. WEKA documentation - CfsSubsetEval. Class CfsSubsetEval. 2016 . Available at: http://weka.sourceforge.net/doc.stable/weka/attributeSelection/CfsSubsetEval.html . Accessed October 10, 2016
• 24 WEKA. WEKA documentation - InfoGainAttributeEval. Class InfoGainAttributeEval. 2016 . Available at: http://weka.sourceforge.net/doc.dev/weka/attributeSelection/InfoGainAttributeEval.html . Accessed October 10, 2016
• 25 Griffin F, Resar R. IHI Global Trigger Tool for Measuring Adverse Events. Second Edition. Cambridge, MA: Institute for Healthcare Improvement; 2009. (IHI Innovation Series white paper)
• 26 Sharek PJ, Parry G, Goldmann D. , et al. Performance characteristics of a methodology to quantify adverse events over time in hospitalized patients. Health Serv Res 2011; 46 (02) 654-678
• 27 Centers for Disease Control and Prevention (CDC). Assessing the National Electronic Injury Surveillance System-Cooperative Adverse Drug Event Surveillance project--six sites, United States, January 1-June 15, 2004. MMWR Morb Mortal Wkly Rep 2005; 54 (15) 380-383
• 28 Karpov A, Parcero C, Mok CPY. , et al. Performance of trigger tools in identifying adverse drug events in emergency department patients: a validation study. Br J Clin Pharmacol 2016; 82 (04) 1048-1057