Online for On Call: A Study Assessing the Use of Internet Resources Including ChatGPT among On-Call Radiology Residents in India

Abstract

Background The information-seeking behavior of the radiology residents on call has undergone modernization in the recent times given the advent of easy to access, reliable online resources, and robust artificial intelligence chatbots such as Chat Generative Pre-Trained Transformer (ChatGPT).

Purpose The aim of this study was to conduct a baseline analysis among the residents to understand the best way to meet information needs in the future, spread awareness about the existing resources, and narrow down to the most preferred online resource.

Methods and Materials A prospective, descriptive study was performed using an online survey instrument and was conducted among radiology residents in India. They were questioned on their demographics, frequency of on call, fatigue experienced on call, and preferred information resources and reasons for choosing them.

Results A total of 286 residents participated in the survey. All residents had used the Internet radiology resources during on-call duties. The most preferred resource material was Radiopaedia followed by Radiology Assistant. IMAIOS e-Anatomy was the most preferred anatomy resource. There was significant (p < 0.05) difference in relation to the use of closed edit peer-reviewed literature among the two batches with it being used almost exclusively by third year residents. In the artificial intelligence-aided ChatGPT section, 61.8% had used the software at least once while being on call, of them 57.6% responded that the information was inaccurate, 67.2% responded that the information was insufficient to aid in diagnosis, 100% felt that the lack of images in the software made it an unlikely resource that would be used by them in the future, and 85.8% agreed that they would use it for providing reporting templates in the future. In the suggestions for upcoming versions, 100% responded that images should be included in the description provide by the chatbot, and 74.5% felt that references for the information being provided should be included as it reaffirms the reliability of the information.

Conclusions Presently, we find that Radiopaedia met most of the requirements as an ideal online radiology resource according to the residents. In the present-day scenario, ChatGPT is not considered as an important on-call radiology education resource first because it lacks images which is quintessential for a budding radiologist, and second, it does not have any reference or proof for the information that it is providing. However, it may be of help to nonmedical professionals who need to understand radiology in layman's terms and to radiologists for patient report preparation and research writing.

Keywords

artificial intelligence - ChatGPT - on-call radiology - resident education

Introduction

Most will agree that our early on-call experiences shape our careers. The indelible purpose of being on-call is being involved; the personalized approach is one that makes it so educationally important. The more on-call a resident does the more competent they tend to be. In some instances, it is an out of the blue diagnostic victory and at times a source of perpetual grief. Nonetheless, there is always something to be learnt even if it is an error that should not be repeated.

The hallmark of resident medical education is seeing a vast number and variety of cases. In the absence of this, acquired knowledge is largely theoretical and unapplied.

Radiologists need to analyze and comment on every image encountered—be it normal or abnormal and if abnormal attempting to describe the abnormality. In a normal functioning, day residents are naturally inclined to look for validation from superiors about each decision they make thereby softening the skill of decision making. Ignoring on-call duty or not executing it efficiently would lead to a generation of radiologists who lack the ability to make decisions and indulge in vague reporting.

Changing Trends in Information Seeking

Radiology, by its very nature, is intrinsically linked to the Internet and is the embodiment of medical technology. Gone are the days where hard bound books and didactic teaching were the only sources of medical education. Internet-based resources are gaining popularity more than ever especially in the context of image interpretation.[1] [2] To add to it the coronavirus disease 2019 (COVID-19) pandemic has reinforced that self-education in the times of social distancing is the only safe alternative. November 2022 saw the launch of a revolutionary new large language model (LLM) Chat Generative Pre-Trained Transformer (ChatGPT) built by OpenAI which intrigued all medical professionals; in March 2023 ChatGPT 4 was launched which has the capability of image interpretation assuming a URL link to the image is provided. Few publications in literature have actually evaluated the available online radiology educational resources and the information-seeking behavior of on-call residents.[3] [4] [5] However, this is the first among residents in Asia and is significant as there has been an increase in the indigenous radiology resources in the recent past; globally this is the first study assessing the utility of ChatGPT among radiology residents.

Methods

An online survey instrument ([Fig. 1], supplementary material) was created using Google Forms (Alphabet Inc., Mountain View, California, United States) and the link to the form (https://forms.gle/fSYpEeXxELmEwS8c6) was circulated via common WhatsApp and Telegram groups having radiology residents. Ethical approval was obtained and the ICJME disclosure form had been filled out. Prior to this a pilot questionnaire was circulated among the residents in our state so as to prefill the probable options in the main survey instrument.

Fig. 1 Survey instrument (section 1).

The link to the Google Form went live on September 26, 2022 and we stopped taking submissions on the April 1, 2023. We received a total of 286 responses. Residents were questioned about their demographic information ([Table 1]), institute/hospital of pursuing residency, and number of residents in the batch/year.

Table 1 Resident demographic characteristics

Yes there is repitition, hence can be removed	n	%
Year of residency
First year	0	0
Second year	116	40.55
Third year	147	51.39
Senior resident	23	8.04
Age y
≤25	2	0.69
26–30	188	65.73
31–35	96	33.56
Sex
Female	102	54.83
Male	184	64.33
Number of residents per batch	n	%
Four	31	10.83
Five	28	9.79
Six	40	13.98
Seven	34	11.88
Eight	91	31.81
Nine	3	1
Ten	25	8.74
Fourteen	17	5.94
Sixteen	17	5.94

The residents were then questioned on the frequency of being on-call ([Fig. 2]) in two separate subheadings namely night duty and 24-hour duties. They were asked about their preferred information resources ([Fig. 3]) during on-call and were given the choices of hardcopy books, softcopy offline books, and the Internet resources.

Fig. 2 Resident attending coverage.

Fig. 3 On-call information-seeking behavior of the residents.

Those responding to using the Internet resources were then questioned on their primary, secondary, and tertiary resources ([Table 2]) used for reference and reasons as to why they chose each of them. They were also questioned on their preferred anatomy resource for referral. A separate section dedicated to ChatGPT was added in December 2022 as it was launched after the forms went live. For all the answers, an option to type in their desired answer was available if they felt their reason was not mentioned among the options. The questions were congruous to a smaller study done in North America[2] that evaluated the on call-seeking behavior among 78 residents. Survey responses were arranged on a spreadsheet, statistically analyzed and results were determined.

Table 2 Resident preferred online resources

Resource	First choice n (%)	Second choice n (%)	Website
Radiopaedia	162 (56.6)	74 (25.87)	https://radiopaedia.org/
Radiology Assistant	44 (15.38)	59 (20.62)	https://radiologyassistant.nl/
Learning Radiology	4 (1.39)	22 (7.69)	http://www.learningradiology.com/
RadioGyan	16 (5.59)	34 (11.89)	https://radiogyan.com/
Radiology Café	0	16 (5.59)	https://www.radiologycafe.com/
Radiology Key	4 (1.39)	12 (4.20)	https://radiologykey.com/
Closed edit peer-reviewed literature
• RadioGraphics	31 (10.83)	41 (14.33)	https://pubs.rsna.org/journal/radiographics
• EPOS Posters	14 (4.89)	14 (4.89)	https://epos.myesr.org/poster/esr
• Eurorad Posters	11 (3.84)	14 (4.90)	https://www.eurorad.org/
Anatomy resources
• IMAIOS e-Anatomy	128 (44.75)	–	https://www.imaios.com/en/e-anatomy
• Radiology Assistant	98 (34.26)	−	https://radiologyassistant.nl/
• W-Radiology	60 (20.97)	–	https://w-radiology.com/

In the ChatGPT section, the residents were questioned whether they ever used the software for radiology information; those that responded were further questioned on what did they use the software for, whether they felt the information was correct, sufficient and what they would want the software to incorporate in upcoming versions to cater to their needs. Residents were also questioned as to what according to them was important for deeming a resource “ideal.”

Results

We received 286 completed forms from residents all over the country which regionally comprised of 75 from institutes of Northern (22—Delhi, 10—Uttar Pradesh, 11—Punjab, 8—Himachal Pradesh, 14—Haryana, 6—Chandigarh, 4—Jammu & Kashmir); 56 from Eastern (19—West Bengal, 21—Odisha, 9—Bihar, 7—Jharkhand); 32 from Western (12—Maharashtra, 7—Gujarat, 13—Rajasthan); 94 from Southern (26—Tamil Nadu, 10—Andhra Pradesh, 9—Kerala, 21—Karnataka, 17—Telangana, 11—Puducherry); 6 from Central (6—Madhya Pradesh); and 23 from North-Eastern (12—Assam, 5—Meghalaya, 2- Manipur, 1- Sikkim, 1- Arunachal Pradesh, 2-Tripura) states and union territories of India, depicted on the map of India ([Fig. 4]).

Fig. 4 Number of responses from each state and union territory.

Of these 116 (40.5%) were second-year residents, 147 (51.3%) were third-year residents, and 23 (8.04%) were senior residents. Majority of the residents 188 (65.73%) were between the age of 26 and 30. Male residents in the study were in majority (184; 64.33%).

Majority (117) of the residents had a night duty once in 8 days (40.9%), while four residents (1.39%) had no night duty. Average frequency of the 24-hour duty for 74 (25.87%) of the residents was once in 3 weeks, and four residents (1.39%) had no 24-hour duty. The hardware with which Internet is usually accessed includes mobile phones, tablets, personal laptops, and department computers.

The on-call information-seeking behavior responses were interesting as 118 (41.25%) of the residents had responded to accessing the Internet for radiology education during call duties at a frequency of 6 to 10 times per 12-hour duty and all 286 residents had used the Internet during on call duties. About 177 (61.88%) of the residents had never used hardcopy radiology books for radiology education during call duties, 120 (41.95%) of the residents had used softcopy radiology books 1 to 5 times per 12-hour duty, and 108 (37.76%) responded to have never used them during on call.

The most popular first choice resources ([Fig. 5]) were as follows: Radiopaedia—162 (56.64%) followed by Radiology Assistant—44 (15.38%). The most popular second-choice resources were as follows: Radiopaedia—74 (25.87%) followed by Radiology Assistant—59 (20.62%). Most residents did not opt for a third-line resource; hence, this category was excluded from evaluation in the study.

Fig. 5 Preferred first- and second-choice resources.

Among the closed edit peer-reviewed literature, RadioGraphics was the most popular with 31 (10.83%) listing it as the preferred first-line resource. Among the online anatomy resources, IMAIOS e-anatomy was the most popular with 128 (44.75%) listing it as their first choice of resource followed by Radiology Assistant—98 (34.26%).

The residents were also asked to rate on a scale of 1 to 5 the feeling of extreme fatigue on call with 1 being the least often and 5 the most often, 110 (38.46%) of the residents responding with 3. The on-call fatigue was least for computed tomography/magnetic resonance imaging and maximum for ultrasonography (USG) especially USG-guided procedures and bedside USG.

We received 612 responses from the residents ([Fig. 6]), for choosing their first-line resource with residents ticking more than one option for the reasons they felt were true.

Fig. 6 Reasoning for first-line resource.

For the second-line resources, the responses were 586 with residents ticking more than one option for the reasons they felt were true was as follows: free of cost—152 (25.93%), free of cost and does not require additional registration – 138 (23.54%), concise and provides adequate information to aid in diagnosis—129 (22.01%), availability of sample cases or sample images—92 (15.69%), up to date with all new classifications, etc.—46 (7.84%), and it has a decent layout for mobile viewing—29 (4.94%).

In the ChatGPT section, 177 (61.88%) had used the software at least once during while being on call, in which 102 (57.62%) responded that the information was inaccurate, 119 (67.23%) responded that the information was insufficient to aid in diagnosis, 177 (100%) responded that the lack of images in the software made it an unlikely resource that they would use in the future, and 152 (85.87%) said that they would use it for providing reporting templates. In the suggestions for upcoming versions, 177 (100%) responded that images should be included in the description provide by the chatbot, and 132 (74.57%) felt that references from where the information was being provided from should be included as it increases the confidence in the information that is provided by the chatbot. This has been summarized in [Fig. 7].

Fig. 7 Summary of Chat Generative Pre-Trained Transformer (ChatGPT) section.

Discussion

We were overwhelmed by the participation of the residents and received a good number of responses given the limited time frame. There were no first-year resident participants in this study as the intake for the year 2022 was delayed due to the COVID-19 pandemic affecting the nationwide counseling and their joining was midway of the data collection period. Essentially our study population consisted of residents with more than a year of experience in the field. All 100% of the residents had used the Internet for reference during on-call duties. The two most preferred resources for both first and second line were Radiopaedia and Radiology Assistant, with 56.6% of the residents listing Radiopaedia as their first-choice resource.

In the pilot survey, all 100% of the residents had listed Google as their first- and second-choice resource; however, Google as an option was removed from the main survey instrument as it is essentially a search engine with no intrinsic radiology information on its website per se; nonetheless it is an invaluable resource based on artificial intelligence that learns to put forth the most frequented sites first on the list of available websites thereby customizing the results for your search query which varies between users despite using the same search keywords. In the pilot questionnaire, few users also responded to having used the reverse image search option (Google Lens) with inconclusive response on the accuracy of the diagnosis. No resident had ever used StatDx that may be due to the fact that it is a paid resource; however, this resource was found to be quite popular in another study conducted in North America.[3]

The game changer Radiopaedia.org, a subsidiary of Radiopaedia (proprietary limited), is a wiki-based international collaborative educational web resource containing a radiology encyclopedia and large repository of sample cases. It is currently the world's largest freely available radiology-related resource with more than 53,326 patient cases and over 16,000 plus reference articles on radiology-related topics as of May 2023.[6] The content is open edit peer-reviewed and can go end number of edits by users as required that is distinct to other peer-reviewed resources in our study. Since its inception in 2005, this resource has gained popularity in particularly low- and middle-income countries that do not have access to paid resources.[6] However, apart from its gratuitous nature its popularity is also largely attributed to its mammoth collection of cases contributed by authors all over the world.

A study published in Journal of the American College of Radiology in 2019[7] found that students who received online education from Radiopaedia.org when compared with standard educational supplements (reading material covering the same content) had a higher mean knowledge score (74%) compared with those who were provided with the traditional learning material (68%) (p = 0.06).

In the upcoming space of artificial intelligence aided resources, ChatGPT has shown promising results for textual information as it is a language-based artificial intelligence and studies have shown its use in patient report preparation and research writing.[8] This is, however, a double-edged sword as with it comes the inevitable issue of copyright infringement as the chatbots generate information that is already on the Internet, in books, and other documents; therefore, it has to be used with caution.

There was statistically significant (p < 0.05) difference among the two batches with second year residents using the Internet with a higher frequency. There was statistically significant (p < 0.05) difference in relation to the use of closed edit peer-reviewed literature among the two batches with it being used almost exclusively by third-year residents. Some of the responses for choosing RadioGraphics as the preferred closed edit peer-reviewed resource was that it had RadioGraphics- Fundamentals and RadioGraphics- On Call subsections with a larger number of articles on radiology at a grassroot level (day-to-day cases) also slide presentations which are simple and suited during the formative years in this subject. We are of the opinion that there are definitely other journals that are excellent resources; however, recall bias also plays a role when only recently read or most frequented journals are in the immediate memory.

No doubt that using the Internet is less time consuming; it also provides instant correlation with a provisional diagnosis, and it is portable and versatile with the option of saving data for discussion with experienced seniors. However, it comes with its own flaws that are its reliability on the strength of Internet connection, choosing an accurate resource is as important and variability of the information from one site to another is also commonly encountered issue.

There were a few limitations in our study that need to be considered. The sample size was relatively small considering the large number of residents in the country; however, the nonavailability of the first-year residents was not controllable. The study is limited by a recall bias wherein the residents have to recall their past on-call experiences in contrast to observing them during on-call. We had no control over the resident demographics as the survey was circulated in various resident groups. There could possibly be a variability in the on-call volume of cases and attending supervision that may have an impact on the information-seeking behavior of the students. Another limitation of the study is that there is no previous available data of the use of Internet among radiology residents in India or on the fatigue of the residents.

Conclusions

Taking all the factors such as limited knowledge, inexperience, and the time constraint that the residents face, using the Internet during on-call duties has been of tremendous value. The most preferred resource was Radiopaedia and the reasons residents chose this resource was its easy accessibility, free of cost, and nonrequirement of additional registration. It provided concise and adequate information to aid in the diagnosis, it had availability of sample cases, it was up to date with all new classifications, and it had an appealing layout for mobile viewing. In the context of our country given the explosion in the software modernization, availability of cheaper hardware, and tech savvy young budding radiologists, the use of the Internet for education in the subject has become the norm. As of today, the importance of ChatGPT as an important artificial intelligence aided radiology educational resource is questionable first because it lacks images that are quintessential for a budding radiologist, and second, as it does not have any reference or proof for the information that it is providing. However, it may be of help to nonmedical professionals who need to understand radiology in layman's terms and to radiologists for patient report preparation and research writing.

The decisive factors while choosing online radiology resources were identified as follows (in order of preference): free resource, requirement of additional registration, concise and provides adequate information to aid in diagnosis, good mobile (smartphone) viewing capability, ability to view sample cases/images, and updated with latest classification and information.

Conflict of Interest

None declared.

Acknowledgments

We would like to thank all the residents who participated in the study.

Ethical Approval and Consent to Participate

Ethical approval and proper consent were taken from all residents participating in the study.

Authors Contributions

This study was directed and coordinated by HSS and SM; HSS, as the principal investigator, provided conceptual and technical guidance for all aspects of the project. SM and HSS planned and executed the study with the help from RD and CM. Analysis of the current resources was done by RD and CM. Literature search for available resources was suggested and executed by RD and HSS. Design of the study was done by CM and HSS. The manuscript was written by HSS and SM and commented on by all authors. All authors have read and approved the manuscript.

• References

• 1 Zafar S, Safdar S, Zafar AN. Evaluation of use of e-Learning in undergraduate radiology education: a review. Eur J Radiol 2014; 83 (12) 2277-2287
• 2 Kok EM, van Geel K, van Merriënboer JJ, Robben SG. What we do and do not know about teaching medical image interpretation. Front Psychol 2017; 8: 309
• 3 Derakhshani A, Ding J, Vijayasarathi A. On-call radiology 2020: where trainees look for help in a high stakes and time sensitive environment. Clin Imaging 2021; 77: 219-223
• 4 Kitchin DR, Applegate KE. Learning radiology a survey investigating radiology resident use of textbooks, journals, and the internet. Acad Radiol 2007; 14 (09) 1113-1120
• 5 D'Alessandro MP, Nguyen BC, D'Alessandro DM. Information needs and information-seeking behaviors of on-call radiology residents. Acad Radiol 1999; 6 (01) 16-21 (Jan)
• 6 Galliard F. Radiopaedia. Accessed May 20, 2023 at: Radiopaedia.org
• 7 El-Ali A, Kamal F, Cabral CL, Squires JH. Comparison of traditional and web based medical student teaching by radiology residents. J Am Coll Radiol 2019; 16 (4 Pt A): 492-495
• 8 Elkassem AA, Smith AD. Potential uses cases for ChatGPT in radiology reporting. Am J Roentgenol 2023; 221 (03) 373-376

Publikationsverlauf

Artikel online veröffentlicht:
21. August 2023