The role of generative language systems in increasing patient awareness of colon cancer screening

Abstract

Background This study aimed to evaluate the effectiveness of ChatGPT (Chat Generative Pretrained Transformer) in answering patientsʼ questions about colorectal cancer (CRC) screening, with the ultimate goal of enhancing patients' awareness and adherence to national screening programs.

Methods 15 questions on CRC screening were posed to ChatGPT4. The answers were rated by 20 gastroenterology experts and 20 nonexperts in three domains (accuracy, completeness, and comprehensibility), and by 100 patients in three dichotomic domains (completeness, comprehensibility, and trustability).

Results According to expert rating, the mean (SD) accuracy score was 4.8 (1.1), on a scale ranging from 1 to 6. The mean (SD) scores for completeness and comprehensibility were 2.1 (0.7) and 2.8 (0.4), respectively, on scales ranging from 1 to 3. Overall, the mean (SD) accuracy (4.8 [1.1] vs. 5.6 [0.7]; P < 0.001) and completeness scores (2.1 [0.7] vs. 2.7 [0.4]; P < 0.001) were significantly lower for the experts than for the nonexperts, while comprehensibility was comparable among the two groups (2.8 [0.4] vs. 2.8 [0.3]; P = 0.55). Patients rated all questions as complete, comprehensible, and trustable in between 97 % and 100 % of cases.

Conclusions ChatGPT shows good performance, with the potential to enhance awareness about CRC and improve screening outcomes. Generative language systems may be further improved after proper training in accordance with scientific evidence and current guidelines.

Publikationsverlauf

Eingereicht: 14. März 2024

Angenommen nach Revision: 14. August 2024

Accepted Manuscript online:
14. August 2024

Artikel online veröffentlicht:
23. Oktober 2024

• References

• 1 Fitzmaurice C, Dicker D, Pain A. et al. Global burden of disease cancer collaboration. The global burden of cancer 2013. JAMA Oncol 2015; 1: 505-527
• 2 Løberg M, Kalager M, Holme Ø. et al. Long-term colorectal-cancer mortality after adenoma removal. NEJM 2014; 371: 799-807
• 3 Bretthauer M, Løberg M, Wieszczy P. et al. Effect of colonoscopy screening on risks of colorectal cancer and related death. NEJM 2022; 387: 1547-1556
• 4 Hewitson P, Glasziou P, Irwig L. et al. Screening for colorectal cancer using the faecal occult blood test, Hemoccult. Cochrane Database Syst Rev 2007; 2007: CD001216
• 5 Tsipa A, O'Connor DB, Branley-Bell D. et al. Promoting colorectal cancer screening: a systematic review and meta-analysis of randomised controlled trials of interventions to increase uptake. Health Psychol Rev 2021; 15: 371-394
• 6 Hahn EE, Baecker A, Shen E. et al. A patient portal-based commitment device to improve adherence with screening for colorectal cancer: a retrospective observational study. J Gen Intern Med 2021; 36: 952-960
• 7 Goshgarian G, Sorourdi C, May FP. et al. Effect of patient portal messaging before mailing fecal immunochemical test kit on colorectal cancer screening rates: a randomized clinical trial. JAMA Netw Open 2022; 5: e2146863
• 8 Klabunde C, Blom J, Bulliard JL. et al. Participation rates for organized colorectal cancer screening programmes: an international comparison. J Med Screen 2015; 22: 119-126
• 9 McNamara D, Leen R, Seng-Lee C. et al. Sustained participation, colonoscopy uptake and adenoma detection rates over two rounds of the Tallaght-Trinity College colorectal cancer screening programme with the faecal immunological test. Eur J Gastroenterol Hepatol 2014; 26: 1415-1421
• 10 Kapidzic A, Grobbee EJ, Hol L. et al. Attendance and yield over three rounds of population-based fecal immuno- chemical test screening. Am J Gastroenterol 2014; 109: 1257-1264
• 11 OpenAI. (2023). ChatGPT (Mar 14 version). Available at (Accessed 5 February 2024): https://chat.openai.com
• 12 Pugliese N, Wai-Sun Wong V, Schattenberg JM. et al. Accuracy, reliability, and comprehensibility of ChatGPT-generated medical responses for patients with nonalcoholic fatty liver disease. Clin Gastroenterol Hepatol 2024; 22: 886-889.e5
• 13 Du RC, Liu X, Lai YK. et al. Exploring the performance of ChatGPT on acute pancreatitis-related questions. J Transl Med 2024; 22: 527
• 14 Lai Y, Liao F, Zhao J. et al. Exploring the capacities of ChatGPT: A comprehensive evaluation of its accuracy and repeatability in addressing helicobacter pylori-related queries. Helicobacter 2024; 29: e13078
• 15 Lee TC, Staller K, Botoman V. et al. ChatGPT answers common patient questions about colonoscopy. Gastroenterology 2023; 165: 509-511.e7
• 16 Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika 1951; 16: 297-334