Download PDF
Appl Clin Inform 2024; 15(05): 1107-1120
DOI: 10.1055/a-2411-5796
Research Article

Enhancing Suicide Attempt Risk Prediction Models with Temporal Clinical Note Features

Kevin J. Krause
1   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Sharon E. Davis
1   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Zhijun Yin
1   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Katherine M. Schafer
1   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Samuel Trent Rosenbloom
1   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Colin G. Walsh
1   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
› Author Affiliations Funding This research has been supported by several funding bodies. The primary source of funding was the National Library of Medicine (NLM) T15 training grant (grant number: 2T15LM007450-20). Additional support came from the Evelyn Selby Stead Fund for Innovation, Vanderbilt University Medical Center, specifically grants R01 MH121455 and R01 MH116269. The Military Suicide Research Consortium also provided funding through grant W81XWH-10-2-0181. Finally, funding for the Research Derivative and BioVU Synthetic Derivative was provided by the National Center for Research Resources (grant number: UL1 RR024975/RR/NCRR). The funders had no role in study design, data collection and analysis, or manuscript preparation.
› Further Information
Also available at
    Permissions and Reprints

Abstract

Objectives The objective of this study was to investigate the impact of enhancing a structured-data-based suicide attempt risk prediction model with temporal Concept Unique Identifiers (CUIs) derived from clinical notes. We aimed to examine how different temporal schemes, model types, and prediction ranges influenced the model's predictive performance. This research sought to improve our understanding of how the integration of temporal information and clinical variable transformation could enhance model predictions.

Methods We identified modeling targets using diagnostic codes for suicide attempts within 30, 90, or 365 days following a temporally grouped visit cluster. Structured data included medications, diagnoses, procedures, and demographics, whereas unstructured data consisted of terms extracted with regular expressions from clinical notes. We compared models trained only on structured data (controls) to hybrid models trained on both structured and unstructured data. We used two temporalization schemes for clinical notes: fixed 90-day windows and flexible epochs. We trained and assessed random forests and hybrid long short-term memory (LSTM) neural networks using area under the precision recall curve (AUPRC) and area under the receiver operating characteristic, with additional evaluation of sensitivity and positive predictive value at 95% specificity.

Results The training set included 2,364,183 visit clusters with 2,009 30-day suicide attempts, and the testing set contained 471,936 visit clusters with 480 suicide attempts. Models trained with temporal CUIs outperformed those trained with only structured data. The window-temporalized LSTM model achieved the highest AUPRC (0.056 ± 0.013) for the 30-day prediction range. Hybrid models generally showed better performance compared with controls across most metrics.

Conclusion This study demonstrated that incorporating electronic health record-derived clinical note features enhanced suicide attempt risk prediction models, particularly with window-temporalized LSTM models. Our results underscored the critical value of unstructured data in suicidality prediction, aligning with previous findings. Future research should focus on integrating more sophisticated methods to continue improving prediction accuracy, which will enhance the effectiveness of future intervention.

Keywords

suicide - machine learning - natural language processing - neural networks - random forest

Protection of Human and Animal Subjects

No human subjects were involved in this project.


Note

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was reviewed by the VUMC Institutional Review Board.


Supplementary Material



Publication History

Received: 22 August 2023

Accepted: 05 September 2024

Accepted Manuscript online:
09 September 2024

Article published online:
18 December 2024

© 2024. Thieme. All rights reserved.

Georg Thieme Verlag KG
Oswald-Hesse-Straße 50, 70469 Stuttgart, Germany

 

  • References

  • 1 Centers for Disease Control and Prevention, National Center for Health Statistics. National Vital Statistics System,. Mortality 2018–2021 on CDC WONDER Online Database; 2021
    PubMed
  • 2 Zalsman G, Hawton K, Wasserman D. et al. Suicide prevention strategies revisited: 10-year systematic review. Lancet Psychiatry 2016; 3 (07) 646-659
    CrossrefPubMedSearch in Google Scholar
  • 3 Mann JJ, Apter A, Bertolote J. et al. Suicide prevention strategies: a systematic review. JAMA 2005; 294 (16) 2064-2074
    CrossrefPubMedSearch in Google Scholar
  • 4 Walsh CG, Johnson KB, Ripperger M. et al. Prospective validation of an electronic health record-based, real-time suicide risk model. JAMA Netw Open 2021; 4 (03) e211428
    CrossrefPubMedSearch in Google Scholar
  • 5 Walsh CG, Ribeiro JD, Franklin JC. Predicting risk of suicide attempts over time through machine learning. Clin Psychol Sci 2017; 5 (03) 457-469
    CrossrefPubMedSearch in Google Scholar
  • 6 Bejan CA, Ripperger M, Wilimitis D. et al. Improving ascertainment of suicidal ideation and suicide attempt with natural language processing. Sci Rep 2022; 12 (01) 15146
    CrossrefPubMedSearch in Google Scholar
  • 7 Young J, Bishop S, Humphrey C, Pavlacic JM. A review of natural language processing in the identification of suicidal behavior. J Affect Disord Rep 2023; 12: 100507
    CrossrefPubMedSearch in Google Scholar
  • 8 Cohen J, Wright-Berryman J, Rohlfs L, Trocinski D, Daniel L, Klatt TW. Integration and validation of a natural language processing machine learning suicide risk prediction model based on open-ended interview language in the emergency department. Front Digit Health 2022; 4: 818705
    CrossrefPubMedSearch in Google Scholar
  • 9 Levis M, Leonard Westgate C, Gui J, Watts BV, Shiner B. Natural language processing of clinical mental health notes may add predictive value to existing suicide risk models. Psychol Med 2021; 51 (08) 1382-1391
    CrossrefPubMedSearch in Google Scholar
  • 10 Coppersmith G, Leary R, Crutchley P, Fine A. Natural language processing of social media as screening for suicide risk. Biomed Inform Insights 2018; 10: 11 78222618792860
    CrossrefPubMedSearch in Google Scholar
  • 11 McCoy Jr TH, Castro VM, Roberson AM, Snapper LA, Perlis RH. Improving prediction of suicide and accidental death after discharge from general hospitals with natural language processing. JAMA Psychiatry 2016; 73 (10) 1064-1071
    CrossrefPubMedSearch in Google Scholar
  • 12 Pestian J, Nasrallah H, Matykiewicz P, Bennett A, Leenaars A. Suicide note classification using natural language processing: a content analysis. Biomed Inform Insights 2010; 2010 (03) 19-28
    PubMedSearch in Google Scholar
  • 13 Tsui FR, Shi L, Ruiz V. et al. Natural language processing and machine learning of electronic health records for prediction of first-time suicide attempts. JAMIA Open 2021; 4 (01) ooab011
    CrossrefPubMedSearch in Google Scholar
  • 14 Meerwijk EL, Tamang SR, Finlay AK, Ilgen MA, Reeves RM, Harris AHS. Suicide theory-guided natural language processing of clinical progress notes to improve prediction of veteran suicide risk: protocol for a mixed-method study. BMJ Open 2022; 12 (08) e065088
    CrossrefPubMedSearch in Google Scholar
  • 15 Ji S. Towards intention understanding in suicidal risk assessment with natural language processing. In: Findings of the Association for Computational Linguistics: EMNLP 2022. Association for Computational Linguistics; 2022 :4028–4038. Accessed September 15, 2024 at: https://aclanthology.org/2022.findings-emnlp.297
    PubMedSearch in Google Scholar
  • 16 Ji S, Yu CP, Fung S fu, Pan S, Long G. Supervised learning for suicidal ideation detection in online user content. Complexity 2018; 2018: 1-10
    PubMedSearch in Google Scholar
  • 17 Arowosegbe A, Oyelade T. Application of natural language processing (NLP) in detecting and preventing suicide ideation: a systematic review. Int J Environ Res Public Health 2023; 20 (02) 1514
    CrossrefPubMedSearch in Google Scholar
  • 18 Zhong QY, Mittal LP, Nathan MD. et al. Use of natural language processing in electronic medical records to identify pregnant women with suicidal behavior: towards a solution to the complex classification problem. Eur J Epidemiol 2019; 34 (02) 153-162
    CrossrefPubMedSearch in Google Scholar
  • 19 Zhang D, Yin C, Zeng J, Yuan X, Zhang P. Combining structured and unstructured data for predictive models: a deep learning approach. BMC Med Inform Decis Mak 2020; 20 (01) 280
    CrossrefPubMedSearch in Google Scholar
  • 20 Thompson K. Programming techniques: regular expression search algorithm. Commun ACM 1968; 11 (06) 419-422
    CrossrefPubMedSearch in Google Scholar
  • 21 Beam AL, Kompa B, Schmaltz A. et al. Clinical concept embeddings learned from massive sources of multimodal medical data. Pac Symp Biocomput 2020; 25: 295-306
    PubMedSearch in Google Scholar
  • 22 Mikolov T, Chen K, Corrado G, Dean J. Efficient estimation of word representations in vector space. ArXiv13013781. Accessed February 13, 2022 at: http://arxiv.org/abs/1301.3781
    PubMed
  • 23 Blei DM, Ng AY, Jordan MI. Latent Dirichlet allocation. J Mach Learn Res 2003; 3 (Jan): 993-1022
    PubMedSearch in Google Scholar
  • 24 Dey L, Haque SKM. Opinion mining from noisy text data. In: Proceedings of the Second Workshop on Analytics for Noisy Unstructured Text Data - AND '08. ACM Press; 2008: 83-90
    PubMedSearch in Google Scholar
  • 25 Turney PD. Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. arXiv:cs/0212032. Accessed February 13, 2022 at: http://arxiv.org/abs/cs/0212032
    PubMed
  • 26 Boggs JM, Quintana LM, Powers JD, Hochberg S, Beck A. Frequency of clinicians' assessments for access to lethal means in persons at risk for suicide. Arch Suicide Res 2022; 26 (01) 127-136
    CrossrefPubMedSearch in Google Scholar
  • 27 Yeskuatov E, Chua SL, Foo LK. Leveraging reddit for suicidal ideation detection: a review of machine learning and natural language processing techniques. Int J Environ Res Public Health 2022; 19 (16) 10347
    CrossrefPubMedSearch in Google Scholar
  • 28 Krause KJ, Shelley J, Becker A, Walsh C. Exploring risk factors in suicidal ideation and attempt concept cooccurrence networks. AMIA Annu Symp Proc 2023; 2022: 644-652
    PubMedSearch in Google Scholar
  • 29 Montesinos López OA, Montesinos López A, Crossa J. Overfitting, model tuning, and evaluation of prediction performance. In: Multivariate Statistical Machine Learning Methods for Genomic Prediction. Springer International Publishing;; 2022: 109-139
    Search in Google Scholar
  • 30 Zhao J, Henriksson A. Learning temporal weights of clinical events using variable importance. BMC Med Inform Decis Mak 2016; 16 (Suppl. 02) 71
    CrossrefPubMedSearch in Google Scholar
  • 31 Zhao J, Henriksson A, Kvist M, Asker L, Boström H. Handling temporality of clinical events for drug safety surveillance. AMIA Annu Symp Proc 2015; 2015: 1371-1380
    PubMedSearch in Google Scholar
  • 32 Singh A, Nadkarni G, Gottesman O, Ellis SB, Bottinger EP, Guttag JV. Incorporating temporal EHR data in predictive models for risk stratification of renal function deterioration. J Biomed Inform 2015; 53: 220-228
    CrossrefPubMedSearch in Google Scholar
  • 33 Choi E, Bahadori MT, Schuetz A, Stewart WF, Sun J, Doctor AI. Doctor AI: predicting clinical events via recurrent neural networks. JMLR Workshop Conf Proc 2016; 56: 301-318
    PubMedSearch in Google Scholar
  • 34 Che Z, Purushotham S, Cho K, Sontag D, Liu Y. Recurrent neural networks for multivariate time series with missing values. Sci Rep 2018; 8 (01) 6085
    CrossrefPubMedSearch in Google Scholar
  • 35 Joiner TE. Why People Die by Suicide. Harvard University Press;; 2005
    Search in Google Scholar
  • 36 Klonsky ED, May AM. The three-step theory (3ST): a new theory of suicide rooted in the “ideation-to-action” framework. Int J Cogn Ther 2015; 8 (02) 114-129
    CrossrefPubMedSearch in Google Scholar
  • 37 Klonsky ED, May AM, Saffer BY. Suicide, suicide attempts, and suicidal ideation. Annu Rev Clin Psychol 2016; 12 (01) 307-330
    CrossrefPubMedSearch in Google Scholar
  • 38 Klonsky ED, Saffer BY, Bryan CJ. Ideation-to-action theories of suicide: a conceptual and empirical update. Curr Opin Psychol 2018; 22: 38-43
    CrossrefPubMedSearch in Google Scholar
  • 39 Van Orden KA, Witte TK, Cukrowicz KC, Braithwaite SR, Selby EA, Joiner Jr TE. The interpersonal theory of suicide. Psychol Rev 2010; 117 (02) 575-600
    CrossrefPubMedSearch in Google Scholar
  • 40 Schafer KM, Kennedy G, Gallyer A, Resnik P. A direct comparison of theory-driven and machine learning prediction of suicide: a meta-analysis. PLoS One 2021; 16 (04) e0249833
    CrossrefPubMedSearch in Google Scholar
  • 41 Walker RL, Shortreed SM, Ziebell RA. et al. Evaluation of electronic health record-based suicide risk prediction models on contemporary data. Appl Clin Inform 2021; 12 (04) 778-787
    Thieme ConnectPubMedSearch in Google Scholar
  • 42 Carter G, Milner A, McGill K, Pirkis J, Kapur N, Spittal MJ. Predicting suicidal behaviours using clinical instruments: systematic review and meta-analysis of positive predictive values for risk scales. Br J Psychiatry 2017; 210 (06) 387-395
    CrossrefPubMedSearch in Google Scholar
  • 43 Wilimitis D, Turer RW, Ripperger M. et al. Integration of face-to-face screening with real-time machine learning to predict risk of suicide among adults. JAMA Netw Open 2022; 5 (05) e2212095
    CrossrefPubMedSearch in Google Scholar
  • 44 McKernan LC, Lenert MC, Crofford LJ, Walsh CG. Outpatient engagement and predicted risk of suicide attempts in fibromyalgia. Arthritis Care Res (Hoboken) 2019; 71 (09) 1255-1263
    CrossrefPubMedSearch in Google Scholar
  • 45 Walsh CG, Ripperger MA, Novak L. et al. Randomized controlled comparative effectiveness trial of risk model-guided clinical decision support for suicide screening. medRxiv 2024
    CrossrefPubMedSearch in Google Scholar
  • 46 Shortreed SM, Walker RL, Johnson E. et al. Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction. NPJ Digit Med 2023; 6 (01) 47
    CrossrefPubMedSearch in Google Scholar
  • 47 Bodenreider O. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res 2004; 32 (Database issue): D267-D270
    CrossrefPubMedSearch in Google Scholar
  • 48 Mandani S, Giuse D, McLemore M, Weitkamp A. Augmenting NLP Results by Leveraging SNOMED CT Relationships for Identification of Implantable Cardiac Devices from Patient Notes. Presented at: SNOMED CT Expo 2019; October 31, 2019; Kuala Lumpur, Malaysia. Accessed September 15, 2024 at: https://confluence.ihtsdotools.org/display/FT/201905+Augmenting+NLP+results+by+leveraging+SNOMED+CT+relationships+for+identification+of+implantable+cardiac+devices+from+patient+notes?preview=/87042613/87043024/201905%20SCT%20Expo%202019%20-%20Madani.pdf
    PubMed
  • 49 Sparck Jones K. A statistical interpretation of term specificity and its application in retrieval. J Doc 1972; 28 (01) 11-21
    CrossrefPubMedSearch in Google Scholar
  • 50 Landauer TK, Dumais ST. A solution to Plato's problem: the latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychol Rev 1997; 104 (02) 211-240
    CrossrefPubMedSearch in Google Scholar
  • 51 Pedregosa F, Varoquaux G, Gramfort A. et al. Scikit-learn: machine learning in Python. J Mach Learn Res 2011; 12: 2825-2830
    PubMedSearch in Google Scholar
  • 52 Paszke A, Gross S, Massa F. et al. PyTorch: an imperative style, high-performance deep learning library. 2019; . Accessed September 15, 2024 at:
    CrossrefPubMedSearch in Google Scholar
  • 53 Platt J. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv Large Margin Classif 2000: 10
    Search in Google Scholar
  • 54 Ross EL, Zuromski KL, Reis BY, Nock MK, Kessler RC, Smoller JW. Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US. JAMA Psychiatry 2021; 78 (06) 642-650
    CrossrefPubMedSearch in Google Scholar
  • 55 Spiegelhalter DJ. Probabilistic prediction in patient management and clinical trials. Stat Med 1986; 5 (05) 421-433
    CrossrefPubMedSearch in Google Scholar
  • 56 Scornet E. Trees, forests, and impurity-based variable importance. ; 2021. Accessed May 16, 2022 at: http://arxiv.org/abs/2001.04295
    PubMed
  • 57 Boggs JM, Beck A, Hubley S. et al. General medical, mental health, and demographic risk factors associated with suicide by firearm compared with other means. Psychiatr Serv 2018; 69 (06) 677-684
    CrossrefPubMedSearch in Google Scholar
  • 58 Pennington J, Socher R, Manning C. Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics; 2014: 1532-1543
    PubMedSearch in Google Scholar
  • 59 Sarsam SM, Al-Samarraie H, Alzahrani AI, Alnumay W, Smith AP. A lexicon-based approach to detecting suicide-related messages on Twitter. Biomed Signal Process Control 2021; 65: 102355
    CrossrefPubMedSearch in Google Scholar
  • 60 Gaur M, Aribandi V, Alambo A. et al. Characterization of time-variant and time-invariant assessment of suicidality on Reddit using C-SSRS. PLoS One 2021; 16 (05) e0250448
    CrossrefPubMedSearch in Google Scholar
  • 61 Cambria E, Li Y, Xing FZ, Poria S, Kwok K. SenticNet 6: ensemble application of symbolic and subsymbolic AI for sentiment analysis. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management. ACM; 2020: 105-114
    PubMedSearch in Google Scholar
  • 62 Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 2002; 16: 321-357
    CrossrefPubMedSearch in Google Scholar
  • 63 Lemaître G, Nogueira F, Aridas CK. Imbalanced-learn: a Python toolbox to tackle the curse of imbalanced datasets in machine learning. J Mach Learn Res 2017; 18 (17) 1-5
    PubMedSearch in Google Scholar