Comparative Evaluation of Artificial Intelligence and Endoscopists' Accuracy in Endoscopic Ultrasound for Identifying Normal Anatomical Structures: A Multi-Institutional Cross-Sectional Study

Aims Endoscopic ultrasound (EUS) has emerged as a powerful diagnostic and therapeutic tool in gastroenterology, providing detailed images of the structure’s layers, as well as nearby structures. However, EUS is operator-dependent and variability in operator proficiency results in discrepancies in diagnostic accuracy. Artificial intelligence (AI) can be trained to process and interpret EUS images in real-time, potentially mitigating the operator-dependent variability and improving diagnostic accuracy. Moreover, AI can analyze vast datasets offering a level of consistency and efficiency that is challenging for human operators to match. The aim of this study is to compare the diagnostic accuracy of an AI-based model (AIWorks-EUS, mdconsgroup, Guayaquil, Ecuador) against that of expert and nonexpert gastrointestinal endoscopists visual impression in the identification of normal anatomical structures during EUS procedures.

Methods A multi-institutional, cross-sectional, comparative study designed to assess the diagnostic accuracy of the AIWorks-EUS (mdconsgroup, Guayaquil, Ecuador), in identifying normal anatomical structures during EUS procedures compared to that of endoscopists of different expertise. Consecutive patient videos of linear-array EUS were included for analysis. Then, the video dataset was reviewed by 9 endoscopists (5 expert endoscopists and 4 nonexpert endoscopists (<250 EUS procedures), who were blinded to the AI model interpretations. Finally, their visual impression was compared to the observation outputs provided by the AI models.

Results A total of 29 normal anatomic structures were evaluated by the endoscopists and the AIWorks-EUS software in 39 videos of linear-array EUS. The software reported an observed agreement (OA) of 100% in 15/29 structures (4/9 mediastinal window, 6/12 in gastric window, 5/8 in duodenal window). Subcarinal space and portal vein (gastric window) obtained the lowest OA (92.35%). The AIWorks-EUS software obtained a pooled sensitivity, specificity, positive (PPV) and negative predictive value (NPV), and OA for anatomical structure recognition of 95.2%, 99.3%, 97.9%, 98.4%, and 98.2%, respectively. Additionally, the software obtained higher diagnostic accuracy than expert and nonexpert endoscopist (P<0.05). Agreement between endoscopists and the AI per organ evaluated was obtained, with a difference of 18% between the organ with higher agreement (Aorta) and the lower agreement (Ampulla). [1] [2] [3]

Conclusions The EUS-based AI model achieved higher diagnostic accuracy than both expert and nonexpert endoscopists in the identification of normal anatomical structures. This model can provide assistance to both groups during live procedures, and studies evaluating its use for training should be conducted to evaluate its application in EUS training programs.

Conflicts of interest

Carlos Robles-Medranda is a key opinion leader and consultant for Pentax Medical, Steris, Medtronic, Motus, Micro-tech, G-Tech Medical Supply, CREO Medical, and mdconsgroup. Issac Raijman is a speaker for Boston Scientific, ConMed, Medtronic, and GI Supplies; advisory board member for Micro-Tech; co-owner of EndoRx. Tyler M. Berzin is a consultant for Medtronic, Boston Scientific, Wision, A.I., Magentiq Eye, and RSIP Vision. The other authors declare no conflicts of interest.

• References

• 1 Robles-Medranda C, Baquerizo-Burgos J, Puga-Tejada M. et al. Development of Convolutional Neural Network Models That Recognize Normal Anatomical Structures During Real-Time Radial- and Linear-Array Endoscopic Ultrasound (with Videos). Gastrointest Endosc 2023; S0016-5107 (23) 02966-8
• 2 Finocchiaro M, Valdivia PCortegoso, Hernansanz A. et al. Training Simulators for Gastrointestinal Endoscopy: Current and Future Perspectives. Cancers (Basel) 2021; 13: 1427
• 3 Han C, Nie C, Shen X. et al. Exploration of an effective training system for the diagnosis of pancreatobiliary diseases with EUS: A prospective study. Endosc Ultrasound 2020; 9 (05) 308-18

Publication History

Article published online:
15 April 2024