Can Natural Language Processing Improve the Efficiency of Vaccine Adverse Event Report Review?^[*]

Summary

Background: Individual case review of spontaneous adverse event (AE) reports remains a cornerstone of medical product safety surveil-lance for industry and regulators. Previously we developed the Vaccine Adverse Event Text Miner (VaeTM) to offer automated information extraction and potentially accelerate the evaluation of large volumes of unstructured data and facilitate signal detection.

Objective: To assess how the information extraction performed by VaeTM impacts the accuracy of a medical expert’s review of the vaccine adverse event report.

Methods: The “outcome of interest” (diagnosis, cause of death, second level diagnosis), “onset time,” and “alternative explanations” (drug, medical and family history) for the adverse event were extracted from 1000 reports from the Vaccine Adverse Event Reporting System (VAERS) using the VaeTM system. We compared the human interpretation, by medical experts, of the VaeTM extracted data with their interpretation of the traditional full text reports for these three variables. Two experienced clinicians alternately reviewed text miner output and full text. A third clinician scored the match rate using a predefined algorithm; the proportion of matches and 95% confidence intervals (CI) were calculated. Review time per report was analyzed.

Results: Proportion of matches between the interpretation of the VaeTM extracted data, compared to the interpretation of the full text: 93% for outcome of interest (95% CI: 91– 94%) and 78% for alternative explanation (95% CI: 75 – 81%). Extracted data on the time to onset was used in 14% of cases and was a match in 54% (95% CI: 46 – 63%) of those cases. When supported by structured time data from reports, the match for time to onset was 79% (95% CI: 76 – 81%). The extracted text averaged 136 (74%) fewer words, resulting in a mean reduction in review time of 50 (58%) seconds per report.

Conclusion: Despite a 74% reduction in words, the clinical conclusion from VaeTM extracted data agreed with the full text in 93% and 78% of reports for the outcome of interest and alternative explanation, respec -tively. The limited amount of extracted time interval data indicates the need for further development of this feature. VaeTM may improve review efficiency, but further study is needed to determine if this level of agreement is sufficient for routine use.

Keywords

Biosurveillance - product surveillance - post-marketing - data mining - natural language processing

^* Supplementary online material published on our website www.methods-online.com

• References

• 1 Varricchio F, Iskander J, Destefano F, Ball R, Pless R, Braun MM, Chen RT. Understanding Vaccine Safety Information from the Vaccine Adverse Event Reporting System (VAERS). Pediatr Infect Dis J 2004; 23: 287-294.
• 2 Vellozzi C, Broder KR, Haber P, Guh A, Nguyen M, Cano M, Lewis P, McNeil MM, Bryant M, Singleton J, Martin D, DeStefano F. Adverse events following influenza A (H1N1) 2009 monovalent vaccines reported to the Vaccine Adverse Event Reporting System, United States, October 1, 2009-January 31, 2010. Vaccine 2010; 45: 7248-7255.
• 3 Centers for Disease Control and Prevention (CDC).. Intussusception among recipients of rotavirus vaccine – United States, 1998–1999. MMWR Morb Mortal Wkly Rep 1999; 48 (Suppl. 27) 577-581.
• 4 Food and Drug Administration.. Guidance for Industry – Good Pharmacovigilance Practices and Pharmacoepidemiologic Assessment. March 2005
• 5 CIOMS.. Practical Aspects of Signal Detection in Pharmacovigilance: Report of CIOMS Working Group VIII. Geneva: 2010
• 6 Woo J, Ball R, Burwen DR, Braun MM. Effects of Stratification on Data Mining in the US Vaccine Adverse Event Reporting System (VAERS). Drug Safety 2008; 31: 667-674.
• 7 Martin D, Menschik D, Bryant-Genevier M, Ball R. Data Mining for Prospective Early Detection of Safety Signals in the Vaccine Adverse Events Reporting System (VAERS): A case study of febrile seizures after a 2010–2011 seasonal influenza virus vaccine. Drug Safety 2013; 36: 547-556.
• 8 Botsis T, Buttolph T, Nguyen MD, Woo EJ, Winiecki S, Ball R. Vaccine Adverse Event Text Mining (VaeTM) System for Extracting Features from Vaccine Safety Reports. J Am Med Inform Assoc 2012; 19: 1011-1018.
• 9 Rüggeberg JU, Gold MS, Bayas JM, Blum MD, Bonhoeffer J, Friedlander S, de Souza Brito G, Heininger U, Imoukhuede B, Khamesipour A, Erlewyn-Lajeunesse M, Martin S, Mäkelä M, Nell P, Pool V, Simpson N. Brighton Collaboration Anaphylaxis Working Group. Anaphylaxis: case definition and guidelines for data collection, analysis, and presentation of immunization safety data. Vaccine 2007; 25 (Suppl. 31) 5675-5684.
• 10 Sejvar JJ, Kohl KS, Gidudu J, Amato A, Bakshi N, Baxter R, Burwen DR, Cornblath DR, Cleerbout J, Edwards KM, Heininger U, Hughes R, KhuriBulos N, Korinthenberg R, Law BJ, Munro U, Maltezou HC, Nell P, Oleske J, Sparks R, Velentgas P, Vermeer P, Wiznitzer M. Brighton Collaboration GBS Working Group. Guillain-Barré syndrome and Fisher syndrome: case definitions and guidelines for collection, analysis, and presentation of immunization safety data. Vaccine 2011; 29 (Suppl. 03) 599-612.
• 11 Botsis T, Woo EJ, Ball R. The contribution of the vaccine adverse event text mining system to the classification of possible Guillain-Barré syndrome reports. Appl Clin Inform 2013; 4 (Suppl. 01) 88-99.
• 12 Botsis T, Ball R. Automating case definitions using literature-based reasoning. Appl Clin Inform 2013; 4 (Suppl. 04) 515-527.
• 13 Demner-Fushman D, Chapman WW, McDonald CJ. What can natural language processing do for clinical decision support?. J Biomed Inform 2009; 42 (Suppl. 05) 760-772.
• 14 Greenes RA. editor. Clinical decision support: the road ahead. Burlington, MA: Elsevier, Inc.; 2007
• 15 Hripcsak G, Austin JH, Alderson PO, Friedman C. Use of natural language processing to translate clinical information from a database of 889,921 chest radiographic reports. Radiology 2002; 224 (Suppl. 01) 157-163.
• 16 Friedman C, Shagina L, Lussier Y, Hripcsak G. Automated encoding of clinical documents based on natural language processing. J Am Med Inform Assoc 2004; 11 (Suppl. 05) 392-402.
• 17 Savova GK, Masanz JJ, Ogren PV, Zheng J, Sohn S, Kipper-Schuler KC, Chute CG. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. J Am Med Inform Assoc 2010; 17 (Suppl. 05) 507-513.
• 18 Coden A, Savova G, Sominsky I, Tanenblatt M, Masanz J, Schuler K, Cooper J, Guan W, de Groen PC. Automatically extracting cancer disease characteristics from pathology reports into a Disease Knowledge Representation Model. J Biomed Inform 2009; 42 (Suppl. 05) 937-949.
• 19 Xu H, Stenner SP, Doan S, Johnson KB, Waitman LR, Denny JC. MedEx: a medication information extraction system for clinical narratives. J Am Med Inform Assoc 2010; 17 (Suppl. 01) 19-24.
• 20 Chapman WW, Bridewell W, Hanbury P, Cooper GF, Buchanan BG. A simple algorithm for identifying negated findings and diseases in discharge summaries. J Biomed Inform 2001; 34 (Suppl. 05) 301-310.
• 21 Harkema H, Dowling JN, Thornblade T, Chapman WW. ConText: an algorithm for determining negation, experiencer, and temporal status from clinical reports. J Biomed Inform 2009; 42 (Suppl. 05) 839-851.
• 22 St-Maurice J, Kuo MH, Gooch P. A proof of concept for assessing emergency room use with primary care data and natural language processing. Methods Inf Med 2013; 52 (Suppl. 01) 33-42.
• 23 Cano C, Blanco A, Peshkin L. Automated identification of diagnosis and co-morbidity in clinical records. Methods Inf Med 2009; 48 (Suppl. 06) 546-551.