The “Flapbot”: A Global Perspective on the Validity and Usability of a Flap Monitoring Chatbot

Abstract

Background The Flapbot chatbot assists in free-flap monitoring, emphasizing accessibility, user-friendliness, and global reliability. This study assesses Flapbot's worldwide validity and usability and uses qualitative analysis to identify areas for future enhancement.

Methods Flapbot, built on Google's DialogFlow, was evaluated by international plastic surgeons. Invitations were sent to the International Lower Limb Reconstruction Collaborative (INTELLECT), International Confederation of Plastic Surgery Societies (ICOPLAST), and the International Microsurgery Club. Out of the 42 surgeons who agreed to participate, 21 tested the Flapbot and completed an online survey on its validity and usability. The survey had 13 validity items and 10 usability items. Data analysis involved computing the individual content validity index (I-CVI) and scale-wide content validity index (S-CVI) for validity, and the system usability score (SUS) for usability. Thematic analysis distilled free-text responses to identify key themes.

Results Nine of 13 items had an I-CVI over 0.78, denoting significant relevance. The S-CVI score stood at 0.82, indicating high relevance. The SUS score was 68, representing average usability. Themes highlighted issues with the current model, development suggestions, and surgeons' concerns regarding growing reliance on digital tools in health care.

Conclusion Flapbot is a promising digital aid for free-flap monitoring. While it showcases notable validity and usability, improvements in functionality, usability, and accessibility are needed for broader global use.

Keywords

Flapbot - Chatbot - free-flap monitoring - natural language processing - digital health

Authors' Contribution

All listed authors contributed to (1) conception and design, acquisition of data, analysis and interpretation of data; (2) drafting the article or revising it critically for important intellectual content; (3) final approval of the version to be published; (4) agreement to be accountable for all aspects of the work.

Institutional Ethical Approval

None. The study was performed in accordance with the Declaration of Helsinki.

^# See Appendix A

Publication History

Received: 24 March 2024

Accepted: 14 June 2024

Accepted Manuscript online:
27 June 2024

Article published online:
31 July 2024

• References

• 1 Knoedler S, Hoch CC, Huelsboemer L. et al. Postoperative free flap monitoring in reconstructive surgery—man or machine?. Front Surg 2023 ;10. Accessed 1 September 2023 at: https://www.frontiersin.org/articles/10.3389/fsurg.2023.1130566
• 2 Ali SR, Dobbs TD, Whitaker IS. Using a ChatBot to support clinical decision-making in free flap monitoring. J Plast Reconstr Aesthet Surg 2022; 75 (07) 2387-2440
• 3 Topol E. The Topol Review. Preparing the Healthcare Workforce to Deliver the Digital Future. 2019: 1-48
• 4 Future of Surgery. Royal College of Surgeons (RCS);. 2018 . Accessed September 19, 2023 at: https://futureofsurgery.rcseng.ac.uk/report-2018/
• 5 Weizenbaum J. ELIZA—a computer program for the study of natural language communication between man and machine. Commun ACM 1966; 9: 36-45
• 6 Tudor Car L, Dhinagaran DA, Kyaw BM. et al. Conversational agents in health care: scoping review and conceptual analysis. J Med Internet Res 2020; 22 (08) e17158
• 7 Milne-Ives M, de Cock C, Lim E. et al. The effectiveness of artificial intelligence conversational agents in health care: systematic review. J Med Internet Res 2020; 22 (10) e20346
• 8 Adamopoulou E, Moussiades L. An overview of Chatbot technology. In: Maglogiannis I, Iliadis L, Pimenidis E. eds. Artificial Intelligence Applications and Innovations. Cham:: Springer International Publishing;; 2020: 373-383
• 9 Denecke K, May R. Usability assessment of conversational agents in healthcare: a literature review. Stud Health Technol Inform 2022; 294: 169-173
• 10 DialogFlow. Google Cloud. Accessed September 1, 2023 at: https://cloud.google.com/dialogflow
• 11 Klug B. An overview of the system usability scale in library website and system usability testing. Weave 2017; 1 (06)
• 12 Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health 2007; 30 (04) 459-467
• 13 Attride-Stirling J. Thematic networks: an analytic tool for qualitative research. Qual Res 2001; 1: 385-405
• 14 Lumivero. NVivo 14. 2020 . Accessed July 10, 2024 at: https://lumivero.com/products/nvivo/
• 15 Open AI. ChatGPT. Accessed July 10, 2024 at: https://openai.com/
• 16 Open AI. Platform. . Accessed September 1, 2023 at: https://platform.openai.com
• 17 Berry CE, Fazilat AZ, Lavin C. et al. Both patients and plastic surgeons prefer artificial intelligence-generated microsurgical information. J Reconstr Microsurg 2024; 26
• 18 Tian WM, Sergesketter AR, Hollenbeck ST. The role of ChatGPT in microsurgery: assessing content quality and potential applications. J Reconstr Microsurg 2024; 40 (03) e1-e2
• 19 OpenAI enables Browse with Bing for live web results in ChatGPT - The Verge. Accessed May 15, 2024 at: https://www.theverge.com/2023/9/27/23892781/openai-chatgpt-live-web-results-browse-with-bing
• 20 Manolitsis I, Feretzakis G, Tzelves L. et al. Training ChatGPT models in assisting urologists in daily practice. Stud Health Technol Inform 2023; 305: 576-579
• 21 Using the UKCA marking. GOV.UK. 2023 . Accessed September 1, 2023 at: https://www.gov.uk/guidance/using-the-ukca-marking
• 22 The Medical Devices Regulations. 2002 . Accessed September 1, 2023 at: https://www.legislation.gov.uk/uksi/2002/618/contents/made
• 23 NHS digital, data and technology standards. NHS Digital. Accessed September 1, 2023 at: https://digital.nhs.uk/about-nhs-digital/corporate-information-and-documents/nhs-digital-data-and-technology-standards
• 24 da Silva Lima Roque G, Roque de Souza R, Araújo do Nascimento JW, de Campos Filho AS, de Melo Queiroz SR, Ramos Vieira Santos IC. Content validation and usability of a Chatbot of guidelines for wound dressing. Int J Med Inform 2021; 151: 104473
• 25 de Campos Filho AS, Vasconcelos Cursino JR, do Nascimento JWA, de Souza RR, da Silva Lima Roque G, de Souza Cavalcanti AR. Content and usability validation of an intelligent virtual conversation assistant used for virtual triage during the COVID-19 pandemic in Brazil. Comput Inform Nurs 2022; 40 (11) 779-785