Use of Natural Language Processing to Identify Sexual and Reproductive Health Information in Clinical Text

Abstract

Objectives This study aimed to enable clinical researchers without expertise in natural language processing (NLP) to extract and analyze information about sexual and reproductive health (SRH), or other sensitive health topics, from large sets of clinical notes.

Methods (1) We retrieved text from the electronic health record as individual notes. (2) We segmented notes into sentences using one of scispaCy's NLP toolkits. (3) We exported sentences to the labeling application Watchful and annotated subsets of these as relevant or irrelevant to various SRH categories by applying a combination of regular expressions and manual annotation. (4) The labeled sentences served as training data to create machine learning models for classifying text; specifically, we used spaCy's default text classification ensemble, comprising a bag-of-words model and a neural network with attention. (5) We applied each model to unlabeled sentences to identify additional references to SRH with novel relevant vocabulary. We used this information and repeated steps 3 to 5 iteratively until the models identified no new relevant sentences for each topic. Finally, we aggregated the labeled data for analysis.

Results This methodology was applied to 3,663 Child Neurology notes for 971 female patients. Our search focused on six SRH categories. We validated the approach using two subject matter experts, who independently labeled a sample of 400 sentences. Cohen's kappa values were calculated for each category between the reviewers (menstruation: 1, sexual activity: 0.9499, contraception: 0.9887, folic acid: 1, teratogens: 0.8864, pregnancy: 0.9499). After removing the sentences on which reviewers did not agree, we compared the reviewers' labels to those produced via our methodology, again using Cohen's kappa (menstruation: 1, sexual activity: 1, contraception: 0.9885, folic acid: 1, teratogens: 0.9841, pregnancy: 0.9871).

Conclusion Our methodology is reproducible, enables analysis of large amounts of text, and has produced results that are highly comparable to subject matter expert manual review.

Keywords

clinical research informatics - machine learning - natural language processing - methodologies - sexual and reproductive health

Note

This work was presented as a poster at the American Epilepsy Society 2022 Annual Meeting.

Publication History

Received: 31 December 2022

Accepted: 19 December 2023

Accepted Manuscript online:
20 December 2023

Article published online:
20 February 2024

• References

• 1 Singh JA, Siddiqi M, Parameshwar P, Chandra-Mouli V. World Health Organization guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health in adolescents. J Adolesc Health 2019; 64 (04) 427-429
• 2 Flicker S, Guta A. Ethical approaches to adolescent participation in sexual health research. J Adolesc Health 2008; 42 (01) 3-10
• 3 Dickson-Swift V, James EL, Kippen S. et al. Doing sensitive research: what challenges do qualitative researchers face?. Qual Res 2007; 7: 327-353
• 4 Vaci N, Liu Q, Kormilitzin A. et al. Natural language processing for structuring clinical text data on depression using UK-CRIS. Evid Based Ment Health 2020; 23 (01) 21-26
• 5 Kirkpatrick L, Collins A, Sogawa Y, Birru Talabi M, Harrison E, Kazmerski TM. Sexual and reproductive healthcare for adolescent and young adult women with epilepsy: a qualitative study of pediatric neurologists and epileptologists. Epilepsy Behav 2020; 104 (Pt A): 106911
• 6 Kirkpatrick L, Harrison E, Khalil S. et al. A survey of child neurologists about reproductive healthcare for adolescent women with epilepsy. Epilepsy Behav 2021; 120: 108001
• 7 Kirkpatrick L, Harrison E, Borrero S. et al. Preferences and experiences of women with epilepsy regarding sexual and reproductive healthcare provision. Epilepsy Behav 2022; 129: 108631
• 8 Pradhan S, Elhadad N, South BR. et al. Evaluating the state of the art in disorder recognition and normalization of the clinical narrative. J Am Med Inform Assoc 2015; 22 (01) 143-154
• 9 Bada M, Eckert M, Evans D. et al. Concept annotation in the CRAFT corpus. BMC Bioinformatics 2012; 13: 161
• 10 Spasic I, Nenadic G. Clinical text data in machine learning: systematic review. JMIR Med Inform 2020; 8 (03) e17984
• 11 Monarch RM. Human-in-the-Loop Machine Learning: Active Learning and Annotation for Human-centered AI. New York: Manning; 2021
• 12 Van Rossum G, Drake Jr FL. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam. 1995. Accessed October 1, 2022, at: https://ir.cwi.nl/pub/5008 https://www.python.org
• 13 Kluyver T, Ragan-Kelley B, Fernando P. et al. Jupyter Notebooks – a publishing format for reproducible computational workflows. In: Positioning and Power in Academic Publishing: Players. Agents and Agendas; . Amsterdam, Netherlands: Fernando Loizides and Birgit Scmidt. IOS Press; 2016: 87-90
• 14 McKinney W. Data structures for statistical computing in python. Proc of the 9th Python in Science Conf. 2010;51–56. Accessed October 1, 2022, at: https://conference.scipy.org/proceedings/scipy2010/pdfs/mckinney.pdf
• 15 Honnibal M, Montani I. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. 2017. Accessed October 1, 2022, at: https://spacy.io/
• 16 Neumann M, King D, Beltagy I. et al. ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing. Paper presented at: Proceedings of the 18th BioNLP Workshop and Shared Task. Association for Computational Linguistics; Italy. 2019;319–327
• 17 Mohanty S, Singleton J. Watchful. San Francisco, CA; 2016. Accessed July 10, 2022, at: https://www.watchful.io/
• 18 Pedregosa F, Varoquaux G, Gramfort A. et al. Scikit-Learn: machine learning in Python. J Mach Learn Res 2011; 12: 2825-2830
• 19 Bickley LS, Szilagyi PG, Bates B. Bates' Guide to Physical Examination and History Taking. Philadelphia: Lippincott Williams & Wilkins; 2007
• 20 Perotte R, Hajicharalambous C, Sugalski G, Underwood JP. Characterization of electronic health record documentation shortcuts: does the use of dotphrases increase efficiency in the Emergency Department?. AMIA Annu Symp Proc 2022; 2021: 969-978
• 21 Mitkov R. The Oxford Handbook of Computational Linguistics. Oxford, UK: Oxford University Press; 2004: 754
• 22 I Harrison E, Kirkpatrick LA, Hochheiser HS, Sogawa Y, Kazmerski TM. A retrospective textual analysis of sexual and reproductive health counseling for adolescent and young adult people with epilepsy of gestational capacity. Epilepsy Behav 2023; 145: 109321
• 23 Yetisgen M, Vanderwende L, Black T. et al. A New Way of Representing Clinical Reports for Rapid Phenotyping. San Francisco, CA: AMIA 2016 Joint Summits on Translational Science; 2016
• 24 Pomares-Quimbaya A, Kreuzthaler M, Schulz S. Current approaches to identify sections within clinical narratives from electronic health records: a systematic review. BMC Med Res Methodol 2019; 19 (01) 155
• 25 Chen Q, Peng Y, Lu Z. BioSentVec: creating sentence embeddings for biomedical texts. Paper presented at: 2019 IEEE International Conference on Healthcare Informatics (ICHI); June 10--13, 2009; Xi'an, China.