Download PDF
CC BY-NC-ND 4.0 · Methods Inf Med
DOI: 10.1055/a-2405-2489
Original Article

Cross-lingual Natural Language Processing on Limited Annotated Case/Radiology Reports in English and Japanese: Insights from the Real-MedNLP Workshop

Shuntaro Yada
1   Graduate School of Science and Technology, Nara Institute of Science and Technology, Nara, Japan
,
Yuta Nakamura
2   22nd Century Medical and Research Center, The University of Tokyo Hospital, Tokyo, Japan
,
Shoko Wakamiya
1   Graduate School of Science and Technology, Nara Institute of Science and Technology, Nara, Japan
,
Eiji Aramaki
1   Graduate School of Science and Technology, Nara Institute of Science and Technology, Nara, Japan
› Author Affiliations Funding This work was supported by JST AIP Trilateral AI Research Grant Number JPMJCR20G9 and MHLW Program Grant Number JPMH21AC500111 (formerly JST AIP-PRISM Grant Number JPMJCR18Y1), Japan.
› Further Information
   Permissions and Reprints

Abstract

Background Textual datasets (corpora) are crucial for the application of natural language processing (NLP) models. However, corpus creation in the medical field is challenging, primarily because of privacy issues with raw clinical data such as health records. Thus, the existing clinical corpora are generally small and scarce. Medical NLP (MedNLP) methodologies perform well with limited data availability.

Objectives We present the outcomes of the Real-MedNLP workshop, which was conducted using limited and parallel medical corpora. Real-MedNLP exhibits three distinct characteristics: (1) limited annotated documents: the training data comprise only a small set (∼100) of case reports (CRs) and radiology reports (RRs) that have been annotated. (2) Bilingually parallel: the constructed corpora are parallel in Japanese and English. (3) Practical tasks: the workshop addresses fundamental tasks, such as named entity recognition (NER) and applied practical tasks.

Methods We propose three tasks: NER of ∼100 available documents (Task 1), NER based only on annotation guidelines for humans (Task 2), and clinical applications (Task 3) consisting of adverse drug effect (ADE) detection for CRs and identical case identification (CI) for RRs.

Results Nine teams participated in this study. The best systems achieved 0.65 and 0.89 F1-scores for CRs and RRs in Task 1, whereas the top scores in Task 2 decreased by 50 to 70%. In Task 3, ADE reports were detected by up to 0.64 F1-score, and CI scored up to 0.96 binary accuracy.

Conclusion Most systems adopt medical-domain–specific pretrained language models using data augmentation methods. Despite the challenge of limited corpus size in Tasks 1 and 2, recent approaches are promising because the partial match scores reached ∼0.8–0.9 F1-scores. Task 3 applications revealed that the different availabilities of external language resources affected the performance per language.

Keywords

natural language processing - machine learning - adverse drug events

Ethical Approval Statement

This study did not require the participants to be involved in any physical or mental intervention. Furthermore, as it did not utilize personally identifiable information, the study was exempt from institutional review board approval in accordance with the Ethical Guidelines for Medical and Health Research Involving Human Subjects outlined by the Japanese national government.




Publication History

Received: 07 June 2023

Accepted: 23 August 2024

Accepted Manuscript online:
29 August 2024

Article published online:
29 October 2024

© 2024. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)

Georg Thieme Verlag KG
Stuttgart · New York

 

  • References

  • 1 Aramaki E, Wakamiya S, Yada S, Nakamura Y. Natural language processing: From bedside to everywhere. Yearb Med Inform 2022; 31 (01) 243-253
    Thieme ConnectPubMedSearch in Google Scholar
  • 2 Suominen H, Salanterä S, Velupillai S. et al. Overview of the ShARe/CLEF eHealth evaluation lab 2013. In: Information Access Evaluation. Multilinguality, Multimodality, and Visualization. Berlin, Heidelberg:: Springer; 2013: 212-231
    Search in Google Scholar
  • 3 Névéol A, Cohen KB, Grouin C. et al. Clinical information extraction at the CLEF eHealth evaluation lab 2016. CEUR Workshop Proc 2016; 1609: 28-42
    PubMedSearch in Google Scholar
  • 4 Henry S, Buchan K, Filannino M, Stubbs A, Uzuner O. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. J Am Med Inform Assoc 2020; 27 (01) 3-12
    CrossrefPubMedSearch in Google Scholar
  • 5 Jagannatha A, Liu F, Liu W, Yu H. Overview of the first natural language processing challenge for extracting medication, indication, and adverse drug events from electronic health record notes (MADE 1.0). Drug Saf 2019; 42 (01) 99-111
    CrossrefPubMedSearch in Google Scholar
  • 6 Ben Abacha A, Mrabet Y, Zhang Y, Shivade C, Langlotz C, Demner-Fushman D. Overview of the MEDIQA 2021 shared task on summarization in the medical domain. In: Proceedings of the 20th Workshop on Biomedical Language Processing. Association for Computational Linguistics; 2021: 74-85
    PubMedSearch in Google Scholar
  • 7 He B, Dong B, Guan Y. et al. Building a comprehensive syntactic and semantic corpus of Chinese clinical texts. J Biomed Inform 2017; 69: 203-217
    CrossrefPubMedSearch in Google Scholar
  • 8 Campillos L, Deléger L, Grouin C, Hamon T, Ligozat AL, Névéol A. A French clinical corpus with comprehensive semantic annotations: development of the medical entity and relation LIMSI annOtated text corpus (MERLOT). Lang Resour Eval 2018; 52 (02) 571-601
    CrossrefPubMedSearch in Google Scholar
  • 9 Oliveira LESE, Peters AC, da Silva AMP. et al. SemClinBr - a multi-institutional and multi-specialty semantically annotated corpus for Portuguese clinical NLP tasks. J Biomed Semantics 2022; 13 (01) 13
    CrossrefPubMedSearch in Google Scholar
  • 10 Morita M, Kano Y, Ohkuma T, Miyabe M, Aramaki E. Overview of the NTCIR-10 MedNLP task. In: Proceedings of the 10th NTCIR Conference on Evaluation of Information Access Technologies, NTCIR-10, National Center of Sciences,. Tokyo. National Institute of Informatics (NII); 2013
    PubMedSearch in Google Scholar
  • 11 Morita M, Kano Y, Ohkuma T, Aramaki E. Overview of the NTCIR-11 MedNLP task. In: Proceedings of the 11th NTCIR Conference on Evaluation of Information Access Technologies, NTCIR-11, National Center of Sciences,. Tokyo. National Institute of Informatics (NII); 2014
    PubMedSearch in Google Scholar
  • 12 Morita M, Kano Y, Ohkuma T, Aramaki E. Overview of the NTCIR-12 MedNLPDoc task. In: Proceedings of the 12th NTCIR Conference on Evaluation of Information Access Technologies, NTCIR-12, National Center of Sciences,. Tokyo. National Institute of Informatics (NII); 2016
    PubMedSearch in Google Scholar
  • 13 Wakamiya S, Morita M, Kano Y, Ohkuma T, Aramaki E. Overview of the NTCIR-13 MedWeb task. In: Proceedings of the 13th NTCIR Conference on Evaluation of Information Access Technologies, NTCIR-13, National Center of Sciences,. Tokyo. National Institute of Informatics (NII); 2017
    PubMedSearch in Google Scholar
  • 14 Uzuner O, Solti I, Xia F, Cadag E. Community annotation experiment for ground truth generation for the i2b2 medication challenge. J Am Med Inform Assoc 2010; 17 (05) 519-523
    CrossrefPubMedSearch in Google Scholar
  • 15 Hedderich MA, Lange L, Adel H, Strötgen J, Klakow D. A survey on recent approaches for natural language processing in low-resource scenarios. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021: 2545-2568
    PubMedSearch in Google Scholar
  • 16 Kirby JC, Speltz P, Rasmussen LV. et al. PheKB: a catalog and workflow for creating electronic phenotype algorithms for transportability. J Am Med Inform Assoc 2016; 23 (06) 1046-1052
    CrossrefPubMedSearch in Google Scholar
  • 17 Xue H, Li J, Xie H, Wang Y. Review of drug repositioning approaches and resources. Int J Biol Sci 2018; 14 (10) 1232-1244
    CrossrefPubMedSearch in Google Scholar
  • 18 Öztürk H, Özgür A, Schwaller P, Laino T, Ozkirimli E. Exploring chemical space using natural language processing methodologies for drug discovery. Drug Discov Today 2020; 25 (04) 689-705
    CrossrefPubMedSearch in Google Scholar
  • 19 Roberts K, Demner-Fushman D, Voorhees EM. et al. Overview of the TREC 2017 precision medicine track. In: Proceedings of the Twenty-Sixth Text REtrieval Conference. 2017: 26
    PubMedSearch in Google Scholar
  • 20 Biswal S, Xiao C, Glass LM, Westover B, Sun J. CLARA: Clinical report auto-completion. In: Proceedings of the Web Conference 2020. WWW '20. Association for Computing Machinery; 2020: 541-550
    PubMedSearch in Google Scholar
  • 21 Yazdani A, Safdari R, Golkar A, RNiakanKalhori S. Words prediction based on N-gram model for free-text entry in electronic health records. Health Inf Sci Syst 2019; 7 (01) 6
    CrossrefPubMedSearch in Google Scholar
  • 22 Pivovarov R, Elhadad N. Automated methods for the summarization of electronic health records. J Am Med Inform Assoc 2015; 22 (05) 938-947
    CrossrefPubMedSearch in Google Scholar
  • 23 J-Stage. Accessed April 18, 2024 at: https://www.jstage.jst.go.jp/
    PubMed
  • 24 Ito K, Nagai H, Okahisa T, Wakamiya S, Iwao T, Aramaki E. J-MeDic: A Japanese disease name dictionary based on real clinical usage. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA); 2018 . Accessed September 4, 2024 at: https://aclanthology.org/L18-1375
    PubMedSearch in Google Scholar
  • 25 Pons E, Braun LM, Hunink MG, Kors JA. Natural language processing in radiology: a systematic review. Radiology 2016; 279 (02) 329-343
    CrossrefPubMedSearch in Google Scholar
  • 26 Nakamura Y, Hanaoka S, Nomura Y. et al. Clinical comparable corpus describing the same subjects with different expressions. Stud Health Technol Inform 2022; 290: 253-257
    PubMedSearch in Google Scholar
  • 27 Yada S, Joh A, Tanaka R, Cheng F, Aramaki E, Kurohashi S. Towards a versatile medical-annotation guideline feasible without heavy medical knowledge: starting from critical lung diseases. In: Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association; 2020 :4565–4572. Accessed September 4, 2024 at: https://www.aclweb.org/anthology/2020.lrec-1.561
    PubMedSearch in Google Scholar
  • 28 Yada S. Aramaki E, Tanaka R, Cheng F, Kurohashi S. Medical/Clinical Text Annotation Guidelines. 2021
    PubMedSearch in Google Scholar
  • 29 Yada S, Tanaka R, Cheng F, Aramaki E, Kurohashi S. Versatile annotation guidelines for clinical-medical text with an application to critical lung diseases [in Japanese]. J Nat Lang Process 2022; 29 (04) 1165-1197
    CrossrefPubMedSearch in Google Scholar
  • 30 Kelly CR, Kunde SS, Khoruts A. Guidance on preparing an investigational new drug application for fecal microbiota transplantation studies. Clin Gastroenterol Hepatol 2014; 12 (02) 283-288
    CrossrefPubMedSearch in Google Scholar
  • 31 NTCIR test collection. Accessed September 4, 2024 at: https://research.nii.ac.jp/ntcir/data/data-en.html
    PubMed
  • 32 Nishiyama T, Nishidani M, Ando A, Yada S, Wakamiya S, Aramaki E. NAISTSOC at the NTCIR-16 Real-MedNLP task. In: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies. 2022: 330-333
    PubMedSearch in Google Scholar
  • 33 Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics; 2019: 4171-4186
    PubMedSearch in Google Scholar
  • 34 BERT models for Japanese NLP. Accessed April 18, 2023 at: https://huggingface.co/cl-tohoku/bert-base-japanese-whole-word-masking
    PubMed
  • 35 MeCab. Accessed April 18, 2023 at: https://taku910.github.io/mecab/
    PubMed
  • 36 MedNER-CR-JA. Accessed April 18, 2023 at: https://huggingface.co/sociocom/MedNER-CR-JA
    PubMed
  • 37 Hiai S, Nagayama S, Kojima A. AMI team at the NTCIR-16 Real-MedNLP task. In: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies;. 2022: 297-304
    PubMedSearch in Google Scholar
  • 38 Zhong Z, Fang L, Cao Y. FRDC at NTCIR-16 Real-MedNLP task. In: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies. 2022: 311-315
    PubMedSearch in Google Scholar
  • 39 Noguchi R. GunNLP at the NTCIR-16 Real-MedNLP task: Collaborative filtering-based similar case identification method via structured data “case matrix.”. In: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies. 2022: 349-352
    PubMedSearch in Google Scholar
  • 40 Ideuchi M, Tsuchiya M, Wang Y, Utiyama M. NICTmed at the NCTIR-16 Real-MedNLP task. In: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies. 2022: 339-344
    PubMedSearch in Google Scholar
  • 41 Shao S, Jin G, Satoh D, Nomura Y. NTTD at the NTCIR-16 Real-MedNLP task. In: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies. 2022: 345-348
    PubMedSearch in Google Scholar
  • 42 Zhang Y, Cheng R, Luo L, Gao H, Jiang S, Dong B. SRCB at the NTCIR-16 Real-MedNLP task. In: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies. 2022: 305-310
    PubMedSearch in Google Scholar
  • 43 Holmes B, Gagorik A, Loving J, Green F, Huang H. Syapse at the NCTIR-16 Real-MedNLP task. In: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies. 2022: 334-338
    PubMedSearch in Google Scholar
  • 44 Fujimoto K, Nishio M, Sugiyama O. et al. Approach for named entity recognition and case identification implemented by ZuKyo-JA sub-team at the NTCIR-16 Real-MedNLP task. In: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies. 2022: 322-329
    PubMedSearch in Google Scholar
  • 45 Nooralahzadeh F, Horvath AN, Krauthammer M. Leveraging token-based concept information and data augmentation in few-resource NER: ZuKyo-EN at the NTCIR-16 Real-MedNLP task. In: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies;. 2022: 316-321
    PubMedSearch in Google Scholar
  • 46 Kawazoe Y, Shibata D, Shinohara E, Aramaki E, Ohe K. A clinical specific BERT developed using a huge Japanese clinical text corpus. PLoS One 2021; 16 (11) e0259763
    CrossrefPubMedSearch in Google Scholar
  • 47 Hyakuyaku-Dictionary. Accessed April 18, 2023 at: https://sociocom.naist.jp/hyakuyaku-dic/
    PubMed
  • 48 ComeJisyo. . Accessed April 18, 2023 at: https://ja.osdn.net/projects/comedic/
    PubMed
  • 49 Lee J, Yoon W, Kim S. et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 2020; 36 (04) 1234-1240
    CrossrefPubMedSearch in Google Scholar
  • 50 Dai X, Adel H. An analysis of simple data augmentation for named entity recognition. In: Proceedings of the 28th International Conference on Computational Linguistics. International Committee on Computational Linguistics;. 2020: 3861-3867
    PubMedSearch in Google Scholar
  • 51 Reimers N, Gurevych I. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics; 2019: 3982-3992
    PubMedSearch in Google Scholar
  • 52 Conneau A, Khandelwal K, Goyal N. et al. Unsupervised cross-lingual representation learning at scale. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics; 2020: 8440-8451
    PubMedSearch in Google Scholar
  • 53 Alsentzer E, Murphy J, Boag W. et al. Publicly available clinical BERT embeddings. In: Proceedings of the 2nd Clinical Natural Language Processing Workshop. Association for Computational Linguistics; 2019: 72-78
    PubMedSearch in Google Scholar
  • 54 Gu Y, Tinn R, Cheng H. et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans Comput Healthc 2021; 3 (01) 1-23
    CrossrefPubMedSearch in Google Scholar
  • 55 Lin C. Miller T, Dligach D, Bethard S, Savova GK. EntityBERT: Entity-centric masking strategy for model pretraining for the clinical domain. In: Proceedings of the 20th Workshop on Biomedical Language Processing. Association for Computational Linguistics; 2021: 191-201
    Search in Google Scholar
  • 56 Zhou B, Cui Q, Wei XS, Chen ZM. BBN: Bilateral-branch network with cumulative learning for long-tailed visual recognition. Published online 2019. doi:
    CrossrefPubMed
  • 57 Aronson AR. Effective mapping of biomedical text to the UMLS metathesaurus: The MetaMap program. In: Proceedings of the AMIA Symposium. American Medical Informatics Association; 2001: 17-21
    PubMedSearch in Google Scholar
  • 58 Neumann M, King D, Beltagy I, Ammar W. ScispaCy: Fast and robust models for biomedical natural language processing. Published online February 2019. Accessed September 4, 2024 at: https://arxiv.org/abs/1902.07669
    PubMed
  • 59 Beltagy I, Lo K, Cohan A. SciBERT: A pretrained language model for scientific text. Published online March 2019. Accessed September 4, 2024 at: https://arxiv.org/abs/1903.10676
    PubMed
  • 60 Brierley JD, Gospodarowicz MK, Wittekind C. TNM Classification of Malignant Tumours. Oxford: John Wiley & Sons; 2017
    Search in Google Scholar
  • 61 Zhuang L, Wayne L, Ya S, Jun Z. A robustly optimized BERT pre-training approach with post-training. In: Proceedings of the 20th Chinese National Conference on Computational Linguistics. Chinese Information Processing Society of China; 2021 :1218–1227. Accessed September 4, 2024 at: https://aclanthology.org/2021.ccl-1.108
    PubMedSearch in Google Scholar
  • 62 Vinh NX, Epps J, Bailey J. Information theoretic measures for clusterings comparison: Is a correction for chance necessary?. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML '09. Association for Computing Machinery; 2009: 1073-1080
    PubMedSearch in Google Scholar
  • 63 Fowlkes EB, Mallows CL. A method for comparing two hierarchical clusterings. J Am Stat Assoc 1983; 78 (383) 553-569
    CrossrefPubMedSearch in Google Scholar
  • 64 ChatGPT. Accessed September 4, 2024 at: https://chat.openai.com/
    PubMed
  • 65 Gemini Team Google. Gemini: A family of highly capable multimodal models. Published online December 2023. Accessed September 4, 2024 at: https://arxiv.org/abs/2312.11805
    PubMed
  • 66 Nakamura Y, Hanaoka S, Yada S, Wakamiya S, Aramaki E. NTCIR-17 MedNLP-SC Radiology Report Subtask overview: Dataset and solutions for automated lung cancer staging. In: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies. 2022: 145-151
    PubMedSearch in Google Scholar
  • 67 Wakamiya S, Pereira LK, Raithel L. et al. NTCIR-17 MedNLP-SC social media adverse drug event detection: Subtask overview. In: Proceedings of the 17th NTCIR Conference on Evaluation of Information Access Technologies. 2022: 131-141
    PubMedSearch in Google Scholar