Medical Information Extraction in the Age of Deep Learning

Summary

Objectives: We survey recent developments in medical Information Extraction (IE) as reported in the literature from the past three years. Our focus is on the fundamental methodological paradigm shift from standard Machine Learning (ML) techniques to Deep Neural Networks (DNNs). We describe applications of this new paradigm concentrating on two basic IE tasks, named entity recognition and relation extraction, for two selected semantic classes—diseases and drugs (or medications)—and relations between them.

Methods: For the time period from 2017 to early 2020, we searched for relevant publications from three major scientific communities: medicine and medical informatics, natural language processing, as well as neural networks and artificial intelligence.

Results: In the past decade, the field of Natural Language Processing (NLP) has undergone a profound methodological shift from symbolic to distributed representations based on the paradigm of Deep Learning (DL). Meanwhile, this trend is, although with some delay, also reflected in the medical NLP community. In the reporting period, overwhelming experimental evidence has been gathered, as illustrated in this survey for medical IE, that DL-based approaches outperform non-DL ones by often large margins. Still, small-sized and access-limited corpora create intrinsic problems for data-greedy DL as do special linguistic phenomena of medical sublanguages that have to be overcome by adaptive learning strategies.

Conclusions: The paradigm shift from (feature-engineered) ML to DNNs changes the fundamental methodological rules of the game for medical NLP. This change is by no means restricted to medical IE but should also deeply influence other areas of medical informatics, either NLP- or non-NLP-based.

Keywords

Neural networks - deep learning - natural language processing - information extraction - named entity recognition - relation extraction

• References

• 1 Goodfellow IJ, Bengio Y, Courville AC. Deep Learning. MIT Press; 2016
• 2 Alom MZ, Taha TM, Yakopcic C, Westberg S, Sidike P, Nasrin MS. , et al. A state-of-the-art survey on deep learning theory and architectures. Electronics 2019; 8 (03) 292
• 3 Pouyanfar S, Sadiq S, Yan Y, Tian H, Tao Y, Presa Reyes ME. et al. A survey on deep learning: algorithms, techniques, and applications. ACM Computing Surveys 2018; 51 (05) 92 (92:1–92:36)
• 4 Goldberg Y. Neural Network Methods for Natural Language Processing. Number 37 in Synthesis Lectures on Human Language Technologies. Morgan & Claypool; 2017.
• 5 Belinkov Y, Glass JR. Analysis methods in neural language processing: a survey. Transactions of the Association for Computational Linguistics 2019; 7: 49-72
• 6 Schmidhuber HJ. Deep learning in neural networks: an overview. Neural Networks 2015; 61: 85-117
• 7 Hohman FM, Kahng M, Pienta R, Chau DH. Visual analytics in deep learning: an interrogative survey for the next frontiers. IEEE Trans Vis Comput Graph 2019; 24 (08) 2674-93
• 8 Nassif AB, Shahin I, Attili I, Azzeh M, Shaalan K. Speech recognition using deep neural networks: a systematic review. IEEE Access 2019; 7: 19143-65
• 9 Young T, Hazarika D, Poria S, Cambria E. Recent trends in deep learning based natural language processing. IEEE Computational Intelligence Magazine 2018; 13 (03) 55-75
• 10 Spasić I, Uzuner Ö, Zhou L. Emerging clinical applications of text analytics. Int J Med Inform 2020; 134: 103974
• 11 Wang Y, Wang L, Rastegar-Mojarad MA, Moon S, Shen F, Afzal N. et al. Clinical information extraction applications: a literature review. J Biomed Inform 2018; 77: 34-49
• 12 Kreimeyer K, Foster M, Pandey A, Arya N, Halford G, Jones SF. et al. Natural language processing systems for capturing and standardizing unstructured clinical information: a systematic review. J Biomed Inform 2017; 73 (Supplement C): 14-29
• 13 Friedman C, Kra P, Rzhetsky A. Two biomedical sublanguages: a description based on the theories of Zellig Harris. J Biomed Inform 2002; 35 (04) 222-35
• 14 Newman-Griffis D, Fosler-Lussier E. Writing habits and telltale neighbors: analyzing clinical concept usage patterns with sublanguage embeddings. In: Proceedings of the 10^th International Workshop on Health Text Mining and Information Analysis LOUHI@EMNLP-IJCNLP 2019. p. 146–56.
• 15 Nunez JJ, Carenini G. Comparing the intrinsic performance of clinical concept embeddings by their field of medicine. In: Proceedings of the 10^th International Workshop on Health Text Mining and Information Analysis LOUHI@EMNLP-IJCNLP 2019. p. 11–7.
• 16 Pyysalo S, Ginter F, Moen H, Salakoski T, Ananiadou S. Distributional semantics resources for biomedical text processing. In: Proceedings of the 5^th International Symposium on Languages in Biology and Medicine, LMB 2013. p. 39–43
• 17 Zhang Y, Chen Q, Yang Z, Lin H, Lu Z. BioWordVec, improving biomedical word embeddings with subword information and MeSH. Scientific Data 2019; 6: 52
• 18 Chen Q, Peng Y, Lu Z. BioSentVec : creating sentence embeddings for biomedical texts. In: Proceedings of the 7^th IEEE International Conference on Healthcare Informatics, ICHI 2019.
• 19 Beltagy I, Lo K, Cohan A. SciBert : a pretrained language model for scientific text. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing & 9^th International Joint Conference on Natural Language Processing EMNLP-IJCNLP 2019. p. 3615–20.
• 20 Lee J, Yoon W, Kim S, Kim D, Kim S, So CH. et al. BioBert : a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 2020; 36 (04) 1234-40
• 21 Alsentzer E, Murphy JR, William Boag W, Weng WH, Jin D, Naumann T. et al. Publicly available clinical Bert embeddings. In: Proceedings of the 2^nd Workshop on Clinical Natural Language Processing ClinicalNLP @ NAACL-HLT 2019. p. 72–8.
• 22 Peng Y, Yan S, Zhiyong Lu Z. Transfer learning in biomedical natural language processing: an evaluation of Bert and ELMo on ten benchmarking datasets. In: Proceedings of the 18^th SIGBioMed Workshop on Biomedical Natural Language Processing and Shared Task BioNLP @ ACL 2019. p. 58–65.
• 23 Conway M, Hu M, Chapman WW. Recent advances in using natural language processing to address public health research questions using social media and consumer-generated data. Yearb Med Inform 2019; 28: 208-17
• 24 Gonzalez-Hernandez G, Sarker A, O’Connor K, Savova GK. Capturing the patient’s perspective: a review of advances in natural language processing of health-related text. Yearb Med Inform 2017; 26: 214-27
• 25 Filannino M, Uzuner Ö. Advancing the state of the art in clinical natural language processing through shared tasks. Yearb Med Inform 2018; 27: 184-92
• 26 Velupillai S, Mowery DL, South BR, Kvist M, Dalianis H. Recent advances in clinical natural language processing in support of semantic analysis. Yearb Med Inform 2015; 24: 183-93
• 27 Meystre SM, Savova GK, Kipper-Schuler KC, Hurdle JF. Extracting information from textual documents in the electronic health record: a review of recent research. Yearb Med Inform 2008; 17: 128-44
• 28 Velupillai S, Suominen H, Liakata M, Roberts A, Shah AD, Morley KI. et al. Using clinical natural language processing for health outcomes research: overview and actionable suggestions for future advances. J Biomed Inform 2018; 88: 11-9
• 29 Wu S, Roberts K, Datta S, Du J, Ji Z, Si Y. et al. Deep learning in clinical natural language processing: a methodical review. J Am Med Inform Assoc 2020; 27 (03) 457-70
• 30 Xiao C, Choi E, Sun J. Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review. J Am Med Inform Assoc 2018; 25 (10) 1419-28
• 31 Shickel B, Tighe PJ, Bihorac A, Rashidi P. Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J Biomed Health Inform 2018; 22 (05) 1589-604
• 32 Miotto R, Wang F, Wang S, Jiang X, Dudley JT. Deep learning for healthcare: review, opportunities and challenges. Brief Bioinform 2017; 19 (06) 1236-46
• 33 Esteva A, Robicquet A, Ramsundar B, Kuleshov V, De Pristo M, Chou K. et al. A guide to deep learning in healthcare. Nat Med 2019; 25 (01) 24-9
• 34 Ching T, Himmelstein DS, Beaulieu-Jones BK, Kalinin AA, Do BT, Way GP. et al. Opportunities and obstacles for deep learning in biology and medicine. J R Soc Interface 2018; 15 (141) 20170387
• 35 Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M. et al Scalable and accurate deep learning for electronic health records. NPJ Digit Med 2018; 1: 18
• 36 Névéol A, Dalianis H, Velupillai S, Savova GK, Zweigenbaum P. Clinical natural language processing in languages other than English: opportunities and challenges. J Biomed Semantics 2018; 9 (01) 12
• 37 Chauhan G, McDermott MBA, Szolovits P. REflex: flexible framework for relation extraction in multiple domains. In: Proceedings of the 18^th SIGBioMed Workshop on Biomedical Natural Language Processing and Shared Task BioNLP @ ACL 2019. p. 30–47.
• 38 Li J, Sun A, Han J, Li C. A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering, page [Early Access]; 2020. Available at: https://arxiv.org/pdf/1812.09449.pdf
• 39 Zhao S, Liu T, Zhao S, Wang F. A neural multi-task learning framework to jointly model medical named entity recognition and normalization. In: Proceedings of the 33^rd AAAI Conference on Artificial Intelligence 2019. p. 817–24.
• 40 Sheikhalishahi S, Miotto R, Dudley JT, Lavelli A, Fabio Rinaldi F, Osmani V. Natural language processing of clinical notes on chronic diseases: systematic review. JMIR Med Inform 2019; 7 (02) e12239
• 41 Koleck TA, Dreisbach C, Bourne PE, Bakken S. Natural language processing of symptoms documented in free-text narratives of electronic health records: a systematic review. J Am Med Inform Assoc 2019; 26 (04) 364-79
• 42 Savova GK, Danciu I, Alamudun F, Miller TA, Lin C, Bitterman DS. et al. Use of natural language processing to extract clinical cancer phenotypes from electronic medical records. Cancer Res 2019; 79 (21) 5463-70
• 43 Datta S, Bernstam EV, Roberts K. A frame semantic overview of NLP-based information extraction for cancer-related EHR notes. J Biomed Inform 2019; 100: 103301
• 44 Li J, Sun Y, Johnson RJ, Sciaky D, Wei CH, Leaman R. et al. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database (Oxford) 2016;2016:baw068.
• 45 Doğan RI, Robert Leaman R, Lu Z. NCBI Disease Corpus: a resource for disease name recognition and concept normalization. J Biomed Inform 2014; 47: 1-10
• 46 Wang X, Zhang Y, Ren X, Zhang Y, Žitnik M, Shang J. et al. Cross-type biomedical named entity recognition with deep multi-task learning. Bioinformatics 2019; 35 (10) 1745-52
• 47 Sachan DS, Xie P, Sachan M, Xing EP. Effective use of bidirectional language modeling for transfer learning in biomedical named entity recognition. In: Proceedings of the [2nd] Conference on Machine Learning for Healthcare 2018. p. 383–402.
• 48 Lou Y, Zhang Y, Qian T, Li F, Xiong S, Ji D. A transition-based joint model for disease named entity recognition and normalization. Bioinformatics 2017; 33 (15) 2363-71
• 49 Xu K, Yang Z, Kang P, Wang Q, Liu W. Document-level attention-based BiLSTM-CRF incorporating disease dictionary for disease named entity recognition. Comput Biol Med 2019; 108: 122-32
• 50 Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed representations of words and phrases and their compositionality. In: Proceedings of the 27^th Annual Conference on Neural Information Processing Systems NIPS 2013. p. 3111–9.
• 51 Hong SK, Lee JG. DTranNER: biomedical named entity recognition with deep learning-based label-label transition model. BMC Bioinformatics 2020; 21 (01) 53
• 52 Peters ME, Neumann M, Iyyer M, Gardner M, Clark CT, Lee K. et al. Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018; 1: Long Papers. p. 2227–37.
• 53 Pennington J, Socher R, Manning CD. GloVe: Global Vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014. p. 1532–43.
• 54 Collobert R, Weston J, Bottou L, Karlen M, Kavukcuoglu K, Kuksa PP. Natural language processing (almost) from scratch. Journal of Machine Learning Research 2011; 12 (76) 2493-537
• 55 He K, Zhang X, Ren S, Sun J. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: Proceedings of the 2015 IEEE International Conference on Computer Vision, ICCV 2015. p. 1026–34.
• 56 Henry S, Buchan K, Filannino M, Stubbs A, Uzuner Ö. 2018 n2c2 Shared Task on Adverse Drug Events and Medication Extraction in Electronic Health Records. J Am Med Inform Assoc 2020; 27 (01) 3-12
• 57 Uzuner Ö, Solt I, Cadag E. Extracting medication information from clinical text. J Am Med Inform Assoc 2010; 17 (05) 514-8
• 58 Johnson AEW, Pollard TJ, Shen L, Lehman LWH, Feng M, Ghassemi MM. et al Mimic-III, a freely accessible critical care database. Scientific Data 2016; 3: 160035
• 59 Jagannatha A, Liu F, Liu W, Yu H. Overview of the First Natural Language Processing Challenge for Extracting Medication, Indication, and Adverse Drug Events from Electronic Health Record Notes (Made 1). Drug Saf 2019; 42 (01) 99-111
• 60 Herrero-Zazo M, Segura-Bedmar I, Martínez P, Declerck T. The DDI corpus: an annotated corpus with pharmacological substances and drug-drug interactions. J Biomed Inform 2013; 46 (05) 914-20
• 61 Segura-Bedmar I, Martínez P, Herrero-Zazo M. SemEval-2013 Task 9: Extraction of Drug-Drug Interactions from Biomedical Texts (DDIExtraction 2013). In: Proceedings of the 7^th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2013. p. 341–50.
• 62 Wei Q, Ji Z, Li Z, Du J, Wang J, Xu J. et al. A study of deep learning approaches for medication and adverse drug event extraction from clinical text. J Am Med Inform Assoc 2019; 27 (01) 13-21
• 63 Gligic L, Kormilitzin A, Goldberg P, Nevado-Holgado A. Named entity recognition in electronic health records using transfer learning bootstrapped neural networks. Neural Netw 2020; 121: 132-9
• 64 Zeng D, Sun C, Lin L, Liu B. LSTM-CRF for drug-named entity recognition. Entropy 2017; 19 (06) 283
• 65 Unanue IJ, Borzeshi EZ, Piccardi M. Recurrent neural networks with specialized word embeddings for health-domain named-entity recognition. J Biomed Inform 2017; 76: 102-9
• 66 Li F, Liu W, Yu H. Extraction of information related to adverse drug events from electronic health record notes: design of an end-to-end model based on deep learning. JMIR Med Inform 2018; 6 (04) e121594
• 67 Wunnava S, Qin X, Kakar T, Sen C, Rundensteiner EA, Kong X. Adverse drug event detection from electronic health records using hierarchical recurrent neural networks with dual-level embedding. Drug Saf 2019; 42 (01) 113-22
• 68 Jagannatha AN, Yu H. Bidirectional RNN for medical event detection in electronic health records. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2016. p. 473–82.
• 69 Jagannatha AN, Yu H. Structured prediction models for RNN based sequence labeling in clinical text. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016. p. 856–65.
• 70 Dandala B, Joopudi V, Devarakonda MV. Adverse drug events detection in clinical notes by jointly modeling entities and relations using neural networks. Drug Saf 2019; 42 (01) 135-46
• 71 Tao C, Filannino M, Uzuner Ö. Prescription extraction using CRFs and word embeddings. J Biomed Inform 2017; 72: 60-6
• 72 Chapman AB, Peterson KS, Alba PR, Du Vall SL, Patterson OV. Detecting adverse drug events with rapidly trained classification models. Drug Saf 2019; 42 (01) 147-56
• 73 Yang X, Bian J, Gong Y, Hogan WR, Wu Y. MADEx: a system for detecting medications, adverse drug events, and their relations from clinical notes. Drug Saf 2019; 42 (01) 123-33
• 74 Christopoulou F, Tran TT, Sahu SK, Miwa M, Ananiadou S. Adverse drug events and medication relation extraction in electronic health records with ensemble deep learning methods. J Am Med Inform Assoc 2020; 27 (01) 39-46
• 75 Sun X, Dong K, Ma L, Sutcliffe RFE, He F, Chen S. et al. Drug-drug interaction extraction via recurrent hybrid convolutional neural networks with an improved focal loss. Entropy 2019; 21 (01) 37
• 76 Zheng W, Lin H, Luo L, Zhao Z, Li Z, Zhang Y. et al An attention-based effective neural model for drug-drug interactions extraction. BMC Bioinformatics 2017; 18: 445
• 77 Wang W, Yang X, Yang C, Guo XW, Zhang X, Wu C. Dependency-based long short term memory network for drug-drug interaction extraction. BMC Bioinformatics 2017; 18 (Supplement 16): 578
• 78 Mikolov T, Chen K, Corrado GS, Dean J. Efficient estimation of word representations in vector space. In: Proceedings of the 1st International Conference on Learning Representations, ICLR 2013.
• 79 Lim S, Lee K, Kang J. Drug drug interaction extraction from the literature using a recursive neural network. PLoS One 2018; 13 (01) e0190926
• 80 Zhang Y, Zheng W, Lin H, Wang J, Yang Z, Dumontier M. Drug-drug interaction extraction via hierarchical RNNs on sequence and shortest dependency paths. Bioinformatics 2018; 34 (05) 828-35
• 81 Raihani A, Laachfoubi N. Extracting drug-drug interactions from biomedical text using a feature-based kernel approach. Journal of Theoretical and Applied Information Technology 2016; 92 (01) 109-20
• 82 Zhang T, Leng J, Liu Y. Deep learning for drug-drug interaction extraction from the literature: a review. Brief Bioinform 2019; bbz087
• 83 Zhang Y, Lin H, Yang Z, Wang J, Su Y, Xu B. et al Neural network-based approaches for biomedical relation classification: a review. J Biomed Inform 2019; 99: 103294
• 84 Vilar S, Friedman C, Hripcsak GM. Detection of drug-drug interactions through data mining studies using clinical sources, scientific literature and social media. Brief Bioinform 2018; 19 (05) 863-77
• 85 Luo Y, Thompson WK, Herr TM, Zeng Z, Berendsen MA, Jonnalagadda SR. et al. Natural language processing for EHR-based pharmacovigilance: a structured review. Drug Saf 2017; 40 (11) 1075-89
• 86 Xu B, Shi X, Zhao Z, Zheng W. Leveraging biomedical resources in Bi-LSTM for drug-drug interaction extraction. IEEE Access 2018; 6: 33432-9
• 87 Dewi IN, Dong S, Hu J. Drug-drug interaction relation extraction with deep convolutional neural networks. In: Proceedings of the 2017 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2017. p. 1795–802.
• 88 Sun X, Ma L, Du X, Feng J, Dong K. Deep convolution neural networks for drug-drug interaction extraction. In: Proceedings of the 2018 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2018. p.1662–8.
• 89 Grave E, Bojanowski P, Gupta P, Joulin A, Mikolov T. Learning word vectors for 157 languages. In: Proceedings of the 11th International Conference on Language Resources and Evaluation, LREC 2018. p. 3483–7.
• 90 Spasić I, Nenadić G. Clinical text data in machine learning: systematic review. JMIR Med Inform 2020; 8 (03) e17984
• 91 Hellrich J, Hahn U. Bad company: neighborhoods in neural embedding spaces considered harmful. In: Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers, COLING 2016. p. 2785–96.
• 92 Wendlandt L, Kummerfeld JK, Mihalcea R. Factors influencing the surprising instability of word embeddings. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018(1): Long Papers. p. 2092–102.
• 93 Diaz GI Fokoue-Nkoutche A, Nannicini G, Samulowitz H. An effective algorithm for hyperparameter optimization of neural networks. IBM Journal of Research and Development 2017; 61 (4-5): 9
• 94 Chiu B, Crichton GKO, Korhonen A, Pyysalo S. How to train good word embeddings for biomedical NLP. In: Proceedings of the 15th Workshop on Biomedical Natural Language Processing BioNLP @ ACL 2016. p. 166–74.
• 95 Kalyan KS, Sangeetha S. SECNLP : a survey of embeddings in clinical natural language processing. J Biomed Inform 2020; 101: 103323
• 96 Khattak FK, Jeblee S, Pou-Prom C, Abdalla M, Meaney C, Rudzicz F. A survey of word embeddings for clinical text. J Biomed Inform 2019; 4: 100057
• 97 Wang Y, Liu S, Afzal N, Rastegar-Mojarad MA, Wang L, Shen F. et al. A comparison of word embeddings for the biomedical natural language processing. J Biomed Inform 2018; 87: 12-20
• 98 Lai S, Liu K, He S, Zhao J. How to generate a good word embedding. IEEE Intelligent Systems 2016; 31 (06) 5-14