PDF icon PDF herunterladen
Open Access
CC BY-NC-ND 4.0 · Indian J Radiol Imaging 2024; 34(02): 276-282
DOI: 10.1055/s-0043-1777746
Original Article

Assessing the Capability of ChatGPT, Google Bard, and Microsoft Bing in Solving Radiology Case Vignettes

Pradosh Kumar Sarangi 
1   Department of Radiodiagnosis, All India Institute of Medical Sciences, Deoghar, Jharkhand, India
,
Ravi Kant Narayan 
2   Department of Anatomy, ESIC Medical College & Hospital, Bihta, Patna, Bihar, India
,
Sudipta Mohakud 
3   Department of Radiodiagnosis, All India Institute of Medical Sciences, Bhubaneswar, Odisha, India
,
Aditi Vats 
3   Department of Radiodiagnosis, All India Institute of Medical Sciences, Bhubaneswar, Odisha, India
,
Debabrata Sahani 
3   Department of Radiodiagnosis, All India Institute of Medical Sciences, Bhubaneswar, Odisha, India
,
Himel Mondal 
4   Department of Physiology, All India Institute of Medical Sciences, Deoghar, Jharkhand, India
› Institutsangaben
Funding None.
› Weitere Informationen
Auch verfügbar aufeRefMedOne
  Lizenzen und Reprints
Preview

Abstract

Background The field of radiology relies on accurate interpretation of medical images for effective diagnosis and patient care. Recent advancements in artificial intelligence (AI) and natural language processing have sparked interest in exploring the potential of AI models in assisting radiologists. However, limited research has been conducted to assess the performance of AI models in radiology case interpretation, particularly in comparison to human experts.

Objective This study aimed to evaluate the performance of ChatGPT, Google Bard, and Bing in solving radiology case vignettes (Fellowship of the Royal College of Radiologists 2A [FRCR2A] examination style questions) by comparing their responses to those provided by two radiology residents.

Methods A total of 120 multiple-choice questions based on radiology case vignettes were formulated according to the pattern of FRCR2A examination. The questions were presented to ChatGPT, Google Bard, and Bing. Two residents wrote the examination with the same questions in 3 hours. The responses generated by the AI models were collected and compared to the answer keys and explanation of the answers was rated by the two radiologists. A cutoff of 60% was set as the passing score.

Results The two residents (63.33 and 57.5%) outperformed the three AI models: Bard (44.17%), Bing (53.33%), and ChatGPT (45%), but only one resident passed the examination. The response patterns among the five respondents were significantly different (p = 0.0117). In addition, the agreement among the generative AI models was significant (intraclass correlation coefficient [ICC] = 0.628), but there was no agreement between the residents (Kappa = –0.376). The explanation of generative AI models in support of answer was 44.72% accurate.

Conclusion Humans exhibited superior accuracy compared to the AI models, showcasing a stronger comprehension of the subject matter. All three AI models included in the study could not achieve the minimum percentage needed to pass an FRCR2A examination. However, generative AI models showed significant agreement in their answers where the residents exhibited low agreement, highlighting a lack of consistency in their responses.

Keywords

artificial intelligence - Bard - Bing - ChatGPT - natural language processing - radiology - FRCR2A - fellowship


Publikationsverlauf

Artikel online veröffentlicht:
29. Dezember 2023

© 2023. Indian Radiological Association. This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)

Thieme Medical and Scientific Publishers Pvt. Ltd.
A-12, 2nd Floor, Sector 2, Noida-201301 UP, India