Subscribe to RSS
DOI: 10.1055/a-2161-1982
The Bethesda ERCP Skills Assessment Tool (BESAT) can reliably differentiate endoscopists of different experience levels
Abstract
Background and study aims The Bethesda ERCP Skill Assessment Tool (BESAT) is a video-based assessment tool of technical endoscopic retrograde cholangiopancreatography (ERCP) skill with previously established validity evidence. We aimed to assess the discriminative validity of the BESAT in differentiating ERCP skill levels.
Methods Twelve experienced ERCP practitioners from tertiary academic centers were asked to blindly rate 43 ERCP videos using the BESAT. ERCP videos consisted of native biliary cannulation and sphincterotomy and were recorded from 10 unique endoscopists of various ERCP experience (from advanced endoscopy fellow to > 10 years of ERCP experience). Inter-rater reliability, discriminative validity, and internal structure validity were subsequently assessed.
Results The BESAT was found to reliably differentiate between endoscopists of varying levels of ERCP experience with experienced ERCPists scoring higher than novice ERCPists in 11 of 13 (85%) instrument items. Inter-rater reliability for BESAT items ranged from good to excellent (intraclass correlation range: 0.86 to 0.93). Internal structure validity was assessed with item-total correlations ranging from 0.53 to 0.83.
Conclusions Study findings demonstrate that the BESAT, a video-based ERCP skill assessment tool, has high inter-rater reliability and has discriminative validity in differentiating novice from expert ERCP skill. Further investigations are needed to determine the role of video-based assessment in improving trainee learning curves and patient outcomes.
Keywords
Pancreatobiliary (ERCP/PTCD) - ERC topics - Training - Quality and logistical aspects - Quality managementPublication History
Received: 20 March 2023
Accepted after revision: 22 August 2023
Accepted Manuscript online:
28 August 2023
Article published online:
28 February 2024
© 2024. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial-License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/).
Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany
-
References
- 1 Vandervoort J, Soetikno RM, Tham TC. et al. Risk factors for complications after performance of ERCP. Gastrointest Endosc 2002; 56: 652-656
- 2 Cotton PB, Garrow DA, Gallagher J. et al. Risk factors for complications after ERCP: a multivariate analysis of 11,497 procedures over 12 years. Gastrointest Endosc 2009; 70: 80-88
- 3 Shahidi N, Ou G, Telford J. et al. Establishing the learning curve for achieving competency in performing colonoscopy: a systematic review. Gastrointest Endosc 2014; 80: 410-416
- 4 Faulx AL, Lightdale JR. ASGE Standards of Practice Committee. et al. Guidelines for privileging, credentialing, and proctoring to perform GI endoscopy. Gastrointest Endosc 2017; 85: 273-281
- 5 Wani S, Keswani R, Hall M. et al. A prospective multicenter study evaluating learning curves and competence in endoscopic ultrasound and endoscopic retrograde cholangiopancreatography among advanced endoscopy trainees: The Rapid Assessment of Trainee Endoscopy Skills Study. Clin Gastroenterol Hepatol 2017; 15: 1758-1767
- 6 Wani S, Hall M, Wang AY. et al. Variation in learning curves and competence for ERCP among advanced endoscopy trainees by using cumulative sum analysis. Gastrointest Endosc 2016; 83: 711-719
- 7 Wani S, Keswani RN, Petersen B. et al. Training in EUS and ERCP: standardizing methods to assess competence. Gastrointest Endosc 2018; 87: 1371-1382
- 8 Wani S, Cote GA, Keswani R. et al. Learning curves for EUS by using cumulative sum analysis: implications for American Society for Gastrointestinal Endoscopy recommendations for training. Gastrointest Endosc 2013; 77: 558-565
- 9 Wani S, Hall M, Keswani RN. et al. Variation in Aptitude of Trainees in Endoscopic Ultrasonography, Based on Cumulative Sum Analysis. Clin Gastroenterol Hepatol 2015; 13: 1318-1325 e1312
- 10 McQueen S, McKinnon V, VanderBeek L. et al. Video-Based Assessment in Surgical Education: A Scoping Review. J Surg Educ 2019; 76: 1645-1654
- 11 Augestad KM, Butt K, Ignjatovic D. et al. Video-based coaching in surgical education: a systematic review and meta-analysis. Surg Endosc 2020; 34: 521-535
- 12 Netter A, Schmitt A, Agostini A. et al. Video-based self-assessment enhances laparoscopic skills on a virtual reality simulator: a randomized controlled trial. Surg Endosc 2021; 35: 6679-6686
- 13 Mota P, Carvalho N, Carvalho-Dias E. et al. Video-based surgical learning: improving trainee education and preparation for surgery. J Surg Educ 2018; 75: 828-835
- 14 Rindos NB, Wroble-Biglan M, Ecker A. et al. Impact of video coaching on gynecologic resident laparoscopic suturing: a randomized controlled trial. J Minim Invasive Gynecol 2017; 24: 426-431
- 15 Elmunzer BJ, Walsh CM, Guiton G. et al. Development and initial validation of an instrument for video-based assessment of technical skill in ERCP. Gastrointest Endosc 2021; 93: 914-923
- 16 Scaffidi MA, Grover SC, Carnahan H. et al. A prospective comparison of live and video-based assessments of colonoscopy performance. Gastrointest Endosc 2018; 87: 766-775
- 17 Jeyalingam T, Walsh CM. Video-based assessments: a promising step in improving polypectomy competency. Gastrointest Endosc 2019; 89: 1231-1233
- 18 Messick S. Validity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. Am Psychologist 1995; 50: 741-749
- 19 Messick S. Educational measurement 1989 3rd ed, p. 13–103.
- 20 Kane M. An argument-based approach to validity. Psychol Bull 1992; 112: 527-535
- 21 Downing S. Reliability: on the reproducability of assessment data. Med Educ 2004;
- 22 Elmunzer BJ, Walsh CM, Guiton G. et al. Development and initial validation of an instrument for video-based assessment of technical skill in ERCP. Gastrointestinal endoscopy 2021; 93: 914-923
- 23 Watkins M. Foundations of clinical research: applications to practice. Pearson Education Inc.. 2009
- 24 Tavares W, Eva KW. Exploring the impact of mental workload on rater-based assessments. Adv Health Sci Educ Theory Pract 2013; 18: 291-303
- 25 Tavares W, Ginsburg S, Eva KW. Selecting and simplifying: rater performance and behavior when considering multiple competencies. Teach Learn Med 2016; 28: 41-51
- 26 Elmunzer BJ, Serrano J, Chak A. et al. Rectal indomethacin alone versus indomethacin and prophylactic pancreatic stent placement for preventing pancreatitis after ERCP: study protocol for a randomized controlled trial. Trials 2016; 17: 120
- 27 Scott DJ, Rege RV, Bergen PC. et al. Measuring operative performance after laparoscopic skills training: edited videotape versus direct observation. J Laparoendosc Adv Surg Tech A 2000; 10: 183-190
- 28 Sawyer JM, Anton NE, Korndorffer JR. et al. Time crunch: increasing the efficiency of assessment of technical surgical skill via brief video clips. Surgery 2018; 163: 933-937