Subscribe to RSS
DOI: 10.1055/a-2147-0571
Direct comparison of multiple computer-aided polyp detection systems
Supported by: Forum Gesundheitsstandort 5409.0–001.01/15Supported by: Bayerisches Zentrum für Krebsforschung
Supported by: Interdisziplinäres Zentrum für Klinische Forschung, Universitätsklinikum Würzburg F-406
Abstract
Background and study aims Artificial intelligence (AI)-based systems for computer-aided detection (CADe) of polyps receive regular updates and occasionally offer customizable detection thresholds, both of which impact their performance, but little is known about these effects. This study aimed to compare the performance of different CADe systems on the same benchmark dataset.
Methods 101 colonoscopy videos were used as benchmark. Each video frame with a visible polyp was manually annotated with bounding boxes, resulting in 129 705 polyp images. The videos were then analyzed by three different CADe systems, representing five conditions: two versions of GI Genius, Endo-AID with detection Types A and B, and EndoMind, a freely available system. Evaluation included an analysis of sensitivity and false-positive rate, among other metrics.
Results Endo-AID detection Type A, the earlier version of GI Genius, and EndoMind detected all 93 polyps. Both the later version of GI Genius and Endo-AID Type B missed 1 polyp. The mean per-frame sensitivities were 50.63 % and 67.85 %, respectively, for the earlier and later versions of GI Genius, 65.60 % and 52.95 %, respectively, for Endo-AID Types A and B, and 60.22 % for EndoMind.
Conclusions This study compares the performance of different CADe systems, different updates, and different configuration modes. This might help clinicians to select the most appropriate system for their specific needs.
Publication History
Received: 03 April 2023
Accepted after revision: 01 August 2023
Accepted Manuscript online:
02 August 2023
Article published online:
05 October 2023
© 2023. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution License, permitting unrestricted use, distribution, and reproduction so long as the original work is properly cited. (https://creativecommons.org/licenses/by/4.0/)
Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany
-
References
- 1 Heresbach D, Barrioz T, Lapalus MG. et al. Miss rate for colorectal neoplastic polyps: a prospective multicenter study of back-to-back video colonoscopies. Endoscopy 2008; 40: 284-290 DOI: 10.1055/s-2007-995618.
- 2 Zhao S, Wang S, Pan P. et al. Magnitude, risk factors, and factors associated with adenoma miss rate of tandem colonoscopy: A systematic review and meta-analysis. Gastroenterology 2019; 156: 1661-1674 e1611 DOI: 10.1053/j.gastro.2019.01.260.
- 3 Kamitani Y, Nonaka K, Isomoto H. Current status and future perspectives of artificial intelligence in colonoscopy. J Clin Med 2022; 11: 2923-2937 DOI: 10.3390/jcm11102923.
- 4 Wang P, Berzin TM, Glissen Brown JR. et al. Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: a prospective randomised controlled study. Gut 2019; 68: 1813-1819 DOI: 10.1136/gutjnl-2018-317500.
- 5 Wang P, Liu P, Glissen Brown JR. et al. Lower adenoma miss rate of computer-aided detection-assisted colonoscopy vs routine white-light colonoscopy in a prospective tandem study. Gastroenterology 2020; 159: 1252-1261 e1255 DOI: 10.1053/j.gastro.2020.06.023.
- 6 Wang P, Liu X, Berzin TM. et al. Effect of a deep-learning computer-aided detection system on adenoma detection during colonoscopy (CADe-DB trial): a double-blind randomised study. Lancet Gastroenterol Hepatol 2020; 5: 343-351 DOI: 10.1016/s2468-1253(19)30411-x.
- 7 Liu P, Wang P, Glissen Brown JR. et al. The single-monitor trial: an embedded CADe system increased adenoma detection during colonoscopy: a prospective randomized study. Therap Adv Gastroenterol 2020; 13 DOI: 10.1177/1756284820979165.
- 8 Su JR, Li Z, Shao XJ. et al. Impact of a real-time automatic quality control system on colorectal polyp and adenoma detection: a prospective randomized controlled study (with videos). Gastrointest Endosc 2020; 91: 415-424 e414 DOI: 10.1016/j.gie.2019.08.026.
- 9 Repici A, Badalamenti M, Maselli R. et al. Efficacy of real-time computer-aided detection of colorectal neoplasia in a randomized trial. Gastroenterology 2020; 159: 512-520 e517 DOI: 10.1053/j.gastro.2020.04.062.
- 10 Gong D, Wu L, Zhang J. et al. Detection of colorectal adenomas with a real-time computer-aided system (ENDOANGEL): a randomised controlled study. Lancet Gastroenterol Hepatol 2020; 5: 352-61 DOI: 10.1016/S2468-1253(19)30413-3.
- 11 Yao L, Zhang L, Liu J. et al. Effect of an artificial intelligence-based quality improvement system on efficacy of a computer-aided detection system in colonoscopy: a four-group parallel study. Endoscopy 2022; 54: 757-768 DOI: 10.1055/a-1706-6174.
- 12 Shaukat A, Lichtenstein DR, Somers SC. et al. Computer-aided detection improves adenomas per colonoscopy for screening and surveillance colonoscopy: A randomized trial. Gastroenterology 2022; 163: 732-741 DOI: 10.1053/j.gastro.2022.05.028.
- 13 Gimeno-Garcia AZ, Negrin DH, Hernandez A. et al. Usefulness of a novel computer-aided detection system for colorectal neoplasia: A randomized controlled trial. Gastrointest Endosc 2022; 97: 528-536 DOI: 10.1016/j.gie.2022.09.029.
- 14 Brand M, Troya J, Krenzer A. et al. Frame-by-frame analysis of a commercially available artificial intelligence polyp detection system in full-length colonoscopies. Digestion 2022; 103: 378-385 DOI: 10.1159/000525345.
- 15 Krenzer A, Makowski K, Hekalo A. et al. Semi-automated machine learning video annotation for gastroenterologists. Stud Health Technol Inform 2021; 281: 484-485 DOI: 10.3233/SHTI210206.
- 16 S3-Leitlinie Kolorektales Karzinom. In: Leitlinienprogramm Onkologie: Deutsche Krebsgesellschaft, Deutsche Krebshilfe, AWMF; 2019. https://www.leitlinienprogramm-onkologie.de/fileadmin/user_upload/Downloads/Leitlinien/Kolorektales_Karzinom/Version_2/LL_KRK_Langversion_2.1.pdf
- 17 Gupta S, Lieberman D, Anderson JC. et al. Recommendations for follow-up after colonoscopy and polypectomy: A consensus update by the U.S. Multi-Society Task Force on Colorectal Cancer. . Gastrointest Endosc 2020; 91: 463-485 e465 DOI: 10.1016/j.gie.2020.01.014.
- 18 Repici A, Spadaccini M, Antonelli G. et al. Artificial intelligence and colonoscopy experience: lessons from two randomised trials. Gut 2022; 71: 757-765 DOI: 10.1136/gutjnl-2021-324471.
- 19 Yamada A, Niikura R, Otani K. et al. Automatic detection of colorectal neoplasia in wireless colon capsule endoscopic images using a deep convolutional neural network. Endoscopy 2021; 53: 832-836 DOI: 10.1055/a-1266-1066.
- 20 Weigt J, Repici A, Antonelli G. et al. Performance of a new integrated computer-assisted system (CADe/CADx) for detection and characterization of colorectal neoplasia. Endoscopy 2021; 54: 180-184 DOI: 10.1055/a-1372-0419.
- 21 Brand M, Troya J, Krenzer A. et al. Development and evaluation of a deep learning model to improve the usability of polyp detection systems during interventions. United Eur Gastroenterol J 2022; 10: 477-484 DOI: 10.1002/ueg2.12235.
- 22 Schauer C, Chieng M, Wang M. et al. Artificial intelligence improves adenoma detection rate during colonoscopy. N Z Med J 2022; 135: 22-30
- 23 Fitting D, Krenzer A, Troya J. et al. A video based benchmark data set (ENDOTEST) to evaluate computer-aided polyp detection systems. Scand J Gastroenterol 2022; 57: 1397-1140 DOI: 10.1080/00365521.2022.2085059.