The Bethesda ERCP Skills Assessment Tool (BESAT) can reliably differentiate endoscopists of different experience levels

Kevin Liu; B. Joseph Elmunzer; Sachin Wani; Tiffany Taft; Catharine M Walsh; Mustafa A Arain; Tyler M. Berzin; James Buxbaum; Christopher DiMaio; Syed M. Abbas Fehmi; Neil Gupta; Sreenivasa Jonnalagadda; Vladimir Kushnir; John T. Maple; Raman Muthusamy; Amit Rastogi; Janak N Shah; Amitabh Chak; Ashley Faulx; Nauzer Forbes; Rajesh N Keswani

doi:10.1055/a-2161-1982

Endoscopy International Open, Inhaltsverzeichnis

CC BY-NC-ND 4.0 · Endosc Int Open 2024; 12(02): E324-E331
DOI: 10.1055/a-2161-1982

Original article

The Bethesda ERCP Skills Assessment Tool (BESAT) can reliably differentiate endoscopists of different experience levels

Kevin Liu

¹Gastroenterology, Banner - University Medical Center Phoenix, Phoenix, United States (Ringgold ID: RIN22386)

,

B. Joseph Elmunzer

²Division of Gastroenterology, Medical University of South Carolina, Charleston, United States

,

Sachin Wani

³Gastroenterology, University of Colorado and Veterans Affairs Medical Center, Aurora, United States

,

Tiffany Taft

⁴Division of Gastroenterology, Northwestern University Feinberg School of Medicine, Chicago, United States

,

Catharine M Walsh

⁵Division of Gastroenterology, Hepatology, and Nutrition and the Research and Learning Institutes, The Hospital for Sick Children, Toronto, Canada (Ringgold ID: RIN7979)

⁶Department of Pediatrics, University of Toronto Faculty of Medicine, Toronto, Canada (Ringgold ID: RIN12366)

⁷The Wilson Centre, University of Toronto, Toronto, Canada

,

Mustafa A Arain

⁸Center for Interventional Endoscopy, AdventHealth Orlando, Orlando, United States (Ringgold ID: RIN440172)

,

Tyler M. Berzin

⁹Gastroenterology, Beth Israel Deaconess Medical Center and Harvard Medical School, Boston, United States

,

James Buxbaum

¹⁰Medicine/Gastroenterology, University of California, San Francisco, San Francisco, United States

,

Christopher DiMaio

¹¹Gastroenterology, Catholic Health St Francis Hospital & Heart Center, Roslyn, United States (Ringgold ID: RIN22471)

,

Syed M. Abbas Fehmi

¹²Internal Medicine, UC San Diego Health System, San Diego, United States

,

Neil Gupta

¹³Gastroenterology, Loyola University Medical Center, Chicago, United States (Ringgold ID: RIN25815)

,

Sreenivasa Jonnalagadda

¹⁴Gastroenterology, Saint Luke's Health System, Kansas City, United States (Ringgold ID: RIN14414)

,

Vladimir Kushnir

¹⁵Gastroenterology, Washington University, St Louis, United States

,

John T. Maple

¹⁶Internal Medicine, University of Oklahoma, Oklahoma City, United States

,

Raman Muthusamy

¹⁷Vatche and Tamar Manoukian Division of Digestive Diseases, University of California, Los Angeles, United States

,

Amit Rastogi

¹⁸Gastroenterology, Kansas University Medical Center, Kansas City, United States

¹⁹Gastroenterology, Veterans Affairs Medical Center, Kansas City, United States

,

Janak N Shah

²⁰Gastroenterology, Ochsner Medical Center - New Orleans, New Orleans, United States (Ringgold ID: RIN81796)

,

Amitabh Chak

²¹Gastroenterology, UH Cleveland Medical Center, Cleveland, United States (Ringgold ID: RIN114516)

,

Ashley Faulx

²¹Gastroenterology, UH Cleveland Medical Center, Cleveland, United States (Ringgold ID: RIN114516)

,

Nauzer Forbes

²²Medicine, University of Calgary, Calgary, Canada

,

Rajesh N Keswani

²³Medicine, Northwestern University Feinberg School of Medicine, Chicago, United States (Ringgold ID: RIN12244)

› Institutsangaben

Abstract

Background and study aims The Bethesda ERCP Skill Assessment Tool (BESAT) is a video-based assessment tool of technical endoscopic retrograde cholangiopancreatography (ERCP) skill with previously established validity evidence. We aimed to assess the discriminative validity of the BESAT in differentiating ERCP skill levels.

Methods Twelve experienced ERCP practitioners from tertiary academic centers were asked to blindly rate 43 ERCP videos using the BESAT. ERCP videos consisted of native biliary cannulation and sphincterotomy and were recorded from 10 unique endoscopists of various ERCP experience (from advanced endoscopy fellow to > 10 years of ERCP experience). Inter-rater reliability, discriminative validity, and internal structure validity were subsequently assessed.

Results The BESAT was found to reliably differentiate between endoscopists of varying levels of ERCP experience with experienced ERCPists scoring higher than novice ERCPists in 11 of 13 (85%) instrument items. Inter-rater reliability for BESAT items ranged from good to excellent (intraclass correlation range: 0.86 to 0.93). Internal structure validity was assessed with item-total correlations ranging from 0.53 to 0.83.

Conclusions Study findings demonstrate that the BESAT, a video-based ERCP skill assessment tool, has high inter-rater reliability and has discriminative validity in differentiating novice from expert ERCP skill. Further investigations are needed to determine the role of video-based assessment in improving trainee learning curves and patient outcomes.

Keywords

Pancreatobiliary (ERCP/PTCD) - ERC topics - Training - Quality and logistical aspects - Quality management

Volltext

Referenzen

References
1 Vandervoort J, Soetikno RM, Tham TC. et al. Risk factors for complications after performance of ERCP. Gastrointest Endosc 2002; 56: 652-656
2 Cotton PB, Garrow DA, Gallagher J. et al. Risk factors for complications after ERCP: a multivariate analysis of 11,497 procedures over 12 years. Gastrointest Endosc 2009; 70: 80-88
3 Shahidi N, Ou G, Telford J. et al. Establishing the learning curve for achieving competency in performing colonoscopy: a systematic review. Gastrointest Endosc 2014; 80: 410-416
4 Faulx AL, Lightdale JR. ASGE Standards of Practice Committee. et al. Guidelines for privileging, credentialing, and proctoring to perform GI endoscopy. Gastrointest Endosc 2017; 85: 273-281
5 Wani S, Keswani R, Hall M. et al. A prospective multicenter study evaluating learning curves and competence in endoscopic ultrasound and endoscopic retrograde cholangiopancreatography among advanced endoscopy trainees: The Rapid Assessment of Trainee Endoscopy Skills Study. Clin Gastroenterol Hepatol 2017; 15: 1758-1767
6 Wani S, Hall M, Wang AY. et al. Variation in learning curves and competence for ERCP among advanced endoscopy trainees by using cumulative sum analysis. Gastrointest Endosc 2016; 83: 711-719
7 Wani S, Keswani RN, Petersen B. et al. Training in EUS and ERCP: standardizing methods to assess competence. Gastrointest Endosc 2018; 87: 1371-1382
8 Wani S, Cote GA, Keswani R. et al. Learning curves for EUS by using cumulative sum analysis: implications for American Society for Gastrointestinal Endoscopy recommendations for training. Gastrointest Endosc 2013; 77: 558-565
9 Wani S, Hall M, Keswani RN. et al. Variation in Aptitude of Trainees in Endoscopic Ultrasonography, Based on Cumulative Sum Analysis. Clin Gastroenterol Hepatol 2015; 13: 1318-1325 e1312
10 McQueen S, McKinnon V, VanderBeek L. et al. Video-Based Assessment in Surgical Education: A Scoping Review. J Surg Educ 2019; 76: 1645-1654
11 Augestad KM, Butt K, Ignjatovic D. et al. Video-based coaching in surgical education: a systematic review and meta-analysis. Surg Endosc 2020; 34: 521-535
12 Netter A, Schmitt A, Agostini A. et al. Video-based self-assessment enhances laparoscopic skills on a virtual reality simulator: a randomized controlled trial. Surg Endosc 2021; 35: 6679-6686
13 Mota P, Carvalho N, Carvalho-Dias E. et al. Video-based surgical learning: improving trainee education and preparation for surgery. J Surg Educ 2018; 75: 828-835
14 Rindos NB, Wroble-Biglan M, Ecker A. et al. Impact of video coaching on gynecologic resident laparoscopic suturing: a randomized controlled trial. J Minim Invasive Gynecol 2017; 24: 426-431
15 Elmunzer BJ, Walsh CM, Guiton G. et al. Development and initial validation of an instrument for video-based assessment of technical skill in ERCP. Gastrointest Endosc 2021; 93: 914-923
16 Scaffidi MA, Grover SC, Carnahan H. et al. A prospective comparison of live and video-based assessments of colonoscopy performance. Gastrointest Endosc 2018; 87: 766-775
17 Jeyalingam T, Walsh CM. Video-based assessments: a promising step in improving polypectomy competency. Gastrointest Endosc 2019; 89: 1231-1233
18 Messick S. Validity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. Am Psychologist 1995; 50: 741-749
19 Messick S. Educational measurement 1989 3rd ed, p. 13–103.
20 Kane M. An argument-based approach to validity. Psychol Bull 1992; 112: 527-535
21 Downing S. Reliability: on the reproducability of assessment data. Med Educ 2004;
22 Elmunzer BJ, Walsh CM, Guiton G. et al. Development and initial validation of an instrument for video-based assessment of technical skill in ERCP. Gastrointestinal endoscopy 2021; 93: 914-923
23 Watkins M. Foundations of clinical research: applications to practice. Pearson Education Inc.. 2009
24 Tavares W, Eva KW. Exploring the impact of mental workload on rater-based assessments. Adv Health Sci Educ Theory Pract 2013; 18: 291-303
25 Tavares W, Ginsburg S, Eva KW. Selecting and simplifying: rater performance and behavior when considering multiple competencies. Teach Learn Med 2016; 28: 41-51
26 Elmunzer BJ, Serrano J, Chak A. et al. Rectal indomethacin alone versus indomethacin and prophylactic pancreatic stent placement for preventing pancreatitis after ERCP: study protocol for a randomized controlled trial. Trials 2016; 17: 120
27 Scott DJ, Rege RV, Bergen PC. et al. Measuring operative performance after laparoscopic skills training: edited videotape versus direct observation. J Laparoendosc Adv Surg Tech A 2000; 10: 183-190
28 Sawyer JM, Anton NE, Korndorffer JR. et al. Time crunch: increasing the efficiency of assessment of technical surgical skill via brief video clips. Surgery 2018; 163: 933-937