Multi-expert annotation of Crohn’s disease images of the small bowel for automatic detection using a convolutional recurrent attention neural network

Abstract

Background and study aims Computer-aided diagnostic tools using deep neural networks are efficient for detection of lesions in endoscopy but require a huge number of images. The impact of the quality of annotation has not been tested yet. Here we describe a multi-expert annotated dataset of images extracted from capsules from Crohn’s disease patients and the impact of the quality of annotations on the accuracy of a recurrent attention neural network.

Methods Images of capsule were annotated by a reader first and then reviewed by three experts in inflammatory bowel disease. Concordance analysis between experts was evaluated by Fleiss’ kappa and all the discordant images were, again, read by all the endoscopists to obtain a consensus annotation. A recurrent attention neural network developed for the study was tested before and after the consensus annotation. Available neural networks (ResNet and VGGNet) were also tested under the same conditions.

Results The final dataset included 3498 images with 2124 non-pathological (60.7 %), 1360 pathological (38.9 %), and 14 (0.4 %) inconclusive. Agreement of the experts was good for distinguishing pathological and non-pathological images with a kappa of 0.79 (P < 0.0001). The accuracy of our classifier and the available neural networks increased after the consensus annotation with a precision of 93.7 %, sensitivity of 93 %, and specificity of 95 %.

Conclusions The accuracy of the neural network increased with improved annotations, suggesting that the number of images needed for the development of these systems could be diminished using a well-designed dataset.

Publication History

Received: 25 November 2020

Accepted: 04 March 2021

Article published online:
21 June 2021

• References

• 1 Iddan G, Meron G, Glukhovsky A. et al. Wireless capsule endoscopy. Nature 2000; 405: 417-417
• 2 Dionisio PM, Gurudu SR, Leighton JA. et al. Capsule endoscopy has a significantly higher diagnostic yield in patients with suspected and established small-bowel crohn’s disease: a meta-analysis. Am J Gastroenterol 2010; 105: 1240-1248
• 3 Böcker U, Dinter D, Litterer C. et al. Comparison of magnetic resonance imaging and video capsule enteroscopy in diagnosing small-bowel pathology: Localization-dependent diagnostic yield. Scand J Gastroenterol 2010; 45: 490-500
• 4 Jensen MD, Nathan T, Rafaelsen SR. et al. Diagnostic accuracy of capsule endoscopy for small bowel crohn’s disease is superior to that of MR enterography or CT enterography. Clin Gastroenterol Hepatol 2011; 9: 124-129
• 5 González-Suárez B, Rodriguez S, Ricart E. et al. Comparison of capsule endoscopy and magnetic resonance enterography for the assessment of small bowel lesions in Crohnʼs disease. Inflamm Bowel Dis 2018; 24: 775-780
• 6 Buisson A, Gonzalez F, Poullenot F. et al. Comparative acceptability and perceived clinical utility of monitoring tools: a nationwide survey of patients with inflammatory bowel disease. Inflamm Bowel Dis 2017; 23: 1425-1433
• 7 Le Berre C, Sandborn WJ, Aridhi S. et al. Application of artificial intelligence to gastroenterology and hepatology. Gastroenterology 2020; 158: 76-94.e2
• 8 Muhammad K, Khan S, Kumar N. et al. Vision-based personalized Wireless Capsule Endoscopy for smart healthcare: Taxonomy, literature review, opportunities and challenges. Future Gen Comp Sys 2020; 113: 266-280
• 9 Yuan Y, Meng MQ-H. Deep learning for polyp recognition in wireless capsule endoscopy images. Med Phys 2017; 44: 1379-1389
• 10 Leenhardt R, Vasseur P, Li C. et al. A neural network algorithm for detection of GI angiectasia during small-bowel capsule endoscopy. Gastrointest Endosc 2019; 89: 189-194
• 11 Aoki T, Yamada A, Aoyama K. et al. Automatic detection of erosions and ulcerations in wireless capsule endoscopy images based on a deep convolutional neural network. Gastrointest Endosc 2019; 89: 357-363
• 12 Fan S, Xu L, Fan Y. et al. Computer-aided detection of small intestinal ulcer and erosion in wireless capsule endoscopy images. Phys Med Biol 2018; 63: 165001
• 13 Alaskar H, Hussain A, Al-Aseem N. et al. Application of convolutional neural networks for automated ulcer detection in wireless capsule endoscopy Images. Sensors (Basel) 2019; 19: 1265
• 14 Klang E, Barash Y, Margalit RY. et al. Deep learning algorithms for automated detection of Crohn's disease ulcers by video capsule endoscopy. Gastrointest Endosc 2020; 91: 606-613
• 15 Leenhardt R, Li C, Le Mouel JP. et al. CAD-CAP: a 25,000-image database serving the development of artificial intelligence for capsule endoscopy. Endosc Int Open 2020; 8: E415-E420
• 16 Ding Z, Shi H, Zhang H. et al. Gastroenterologist-level identification of small-bowel diseases and normal variants by capsule endoscopy using a deep-learning model. Gastroenterology 2019; 157: 1044-1054
• 17 Wang S, Xing Y, Zhang L. et al. Deep convolutional neural network for ulcer recognition in wireless capsule endoscopy: experimental feasibility and optimization. Comput Math Methods Med 2019; 2019: 7546215
• 18 Leenhardt R, Buisson A, Bourreille A. et al. Nomenclature and semantic descriptions of ulcerative and inflammatory lesions seen in Crohnʼs disease in small bowel capsule endoscopy: An international Delphi consensus statement. United European Gastroenterol J 2020; 8: 99-107
• 19 Gralnek IM, Defranchis R, Seidman E. et al. Development of a capsule endoscopy scoring index for small bowel mucosal inflammatory change: development of a capsule endoscopy scoring index. Aliment Pharmacol Ther 2007; 27: 146-154
• 20 Vallée R, de Maissin A, Coutrot A. Accurate small bowel lesions detection in wireless capsule endoscopy images using deep recurrent attention neural network. IEEE 21st International Workshop on Multimedia Signal Processing (MMSP 2019), Sep 2019, Kuala Lumpur, Malaysia.
• 21 Vallée R, de Maissin A, Coutrot A. CrohnIPI: An endoscopic image database for the evaluation of automatic Crohn's disease lesions recognition algorithms. Proc SPIE 2020; 11317
• 22 Deng J, Dong W, Socher R. et al. ImageNet: A large-scale hierarchical image database. 2009 IEEE Conf Comput Vis Pattern Recognit Miami, FL: IEEE; 2009. 248-255
• 23 Cho K, van Merrienboer B, Bahdanau D. et al. On the properties of neural machine translation: encoder-decoder approaches. ArXiv14091259 Cs Stat 2014.
• 24 Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull 1971; 76: 378-382
• 25 Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. ICLR – 2015 conference paper. ArXiv: 1409.1556v6.
• 26 Kaiming He, Xiangyu Z, Shaoqing R. et al. Deep residual learning for image recognition. ILSVRC 2015 conference paper. ArXiv: 1512.03385v1.