Influence of video-based feedback on self-assessment accuracy of endoscopic skills: a randomized controlled trial^[*]

Abstract

Background and study aims Novice endoscopists are inaccurate in self-assessment of procedures. One means of improving self-assessment accuracy is through video-based feedback. We aimed to determine the comparative effectiveness of three video-based interventions on novice endoscopists’ self-assessment accuracy of endoscopic competence.

Materials and methods Novice endoscopists (performed < 20 previous procedures) were recruited. Participants completed a simulated esophagogastroduodenoscopy (EGD) on a virtual reality simulator. They were then randomized to one of three groups: self-video review (SVR), which involved watching a recorded video of their own performance; benchmark review (BVR), which involved watching a video of a simulated EGD completed by an expert; and self- and benchmark video (SBVR), which involved both videos. Participants then completed two additional simulated EGD cases. Self-assessments were conducted immediately after the first procedure, after the video intervention and after the additional two procedures. External assessments were conducted by two experienced endoscopists, who were blinded to participant identity and group assignment through video recordings. External and self-assessments were completed using the global rating scale component of the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT GRS).

Results Fifty-one participants completed the study. The BVR group had significantly improved self-assessment accuracy in the short-term, compared to the SBVR group (P = .005). The SBVR group demonstrated significantly improved self-assessment accuracy over time (P = .016). There were no significant effects of group or of time for the SVR group.

Conclusions Video-based interventions, particularly combined use of self- and benchmark video review, can improve accuracy of self-assessment of endoscopic competence among novices.

• References

• 1 Ekkelenkamp VE, Koch AD, Haringsma J. et al. Quality evaluation through self-assessment: a novel method to gain insight into ERCP performance. Frontline Gastroenterol 2014; 5: 10-16
• 2 Koch AD, Haringsma J, Schoon EJ. et al. Competence measurement during colonoscopy training: the use of self-assessment of performance measures. Am J Gastroenterol 2012; 107: 971-975
• 3 Joint Advisory Group on Gastrointestinal Endoscopy. Joint Advisory Group on Gastrointestinal Endoscopy (JAG) accreditation standards for endoscopy services. 2014 Available at https://www.thejag.org.uk/downloads/Accreditation/JAG%20accreditation%20standards%20for%20endoscopy%20services.pdf [Accessed February 26, 2019]
• 4 American Society of Gastrointestinal Endoscopy. Gastrointestinal Endoscopy Self-Assessment Program VIII (GESAP VIII).
• 5 Ansell J, Hurley JJ, Horwood J. et al. Can endoscopists accurately self-assess performance during simulated colonoscopic polypectomy? A prospective, cross-sectional study. Am J Surg 2014; 207: 32-38
• 6 Scaffidi MA, Grover SC, Carnahan H. et al. Impact of experience on self-assessment accuracy of clinical colonoscopy competence. Gastrointest Endosc 2018; 87: 827-836
• 7 Moritz V, Holme O, Leblanc M. et al. An explorative study from the Norwegian Quality Register Gastronet comparing self-estimated versus registered quality in colonoscopy performance. Endosc Int Open 2016; 4: E326-E332
• 8 Vyasa P, Willis RE, Dunkin BJ. et al. Are general surgery residents accurate assessors of their own flexible endoscopy skills?. J Surg Educ 2016; 74: 23-29
• 9 Arora S, Miskovic D, Hull L. et al. Self vs expert assessment of technical and non-technical skills in high fidelity simulation. Am J Surg 2011; 202: 500-506
• 10 Brewster LP, Risucci DA, Joehl RJ. et al. Comparison of resident self-assessments with trained faculty and standardized patient assessments of clinical and technical skills in a structured educational module. Am J Surg 2008; 195: 1-4
• 11 Ward M, MacRae H, Schlachta C. et al. Resident self-assessment of operative performance. Am J Surg 2003; 185: 521-524
• 12 Hawkins SC, Osborne A, Schofield SJ. et al. Improving the accuracy of self-assessment of practical clinical skills using video feedback The importance of including benchmarks. Med Teach 2012; 34: 279-284
• 13 Cheng A, Kessler D, Mackinnon R. et al. Reporting guidelines for health care simulation research: extensions to the CONSORT and STROBE statements. Adv Simul 2016; 1: 25
• 14 Ekkelenkamp VE, Koch AD, De Man RA. et al. Training and competence assessment in GI endoscopy: a systematic review. Gut 2016; 65: 607-615
• 15 Triantafyllou K, Lazaridis LD, Dimitriadis GD. Virtual reality simulators for gastrointestinal endoscopy training. World J Gastrointest Endosc 2014; 6: 6-12
• 16 Walsh CM, Ling SC, Walters TD. et al. Development of the Gastrointestinal Endoscopy Competency Assessment Tool for Pediatric Colonoscopy (GiECATKIDS). J Pediatr Gastroenterol Nutr 2014; 59: 480-486
• 17 Walsh CM, Ling SC, Khanna N. et al. Gastrointestinal endoscopy competency assessment tool: Reliability and validity evidence. Gastrointest Endosc 2015; 81: 1417-1424.e2
• 18 Grover SC, Garg A, Scaffidi MA. et al. Impact of a simulation training curriculum on technical and nontechnical skills in colonoscopy: a randomized trial. Gastrointest Endosc 2015; 82: 1072-1079
• 19 Grover SC, Scaffidi MA, Khan R. et al. Progressive learning in endoscopy simulation training improves clinical performance: a blinded randomized trial. Gastrointest Endosc 2017; 86: 881-889
• 20 Ilgen JS, Ma IWY, Hatala R. et al. A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Med Educ 2015; 49: 161-173
• 21 Scaffidi MA, Grover SC, Carnahan H. et al. A prospective comparison of live and video-based assessments of colonoscopy performance. Gastrointest Endosc 2016; 83: AB116
• 22 Watson PF, Petrie A. Method agreement analysis: A review of correct methodology. Theriogenology 2010; 73: 1167-1179
• 23 Ludbrook J. Confidence in Altman-Bland plots: A critical review of the method of differences. Clin Exp Pharmacol Physiol 2010; 37: 143-149
• 24 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159-174
• 25 Dunn OJ. Multiple comparisons using rank sums American. Technometrics 1964; 6: 241-252
• 26 Tomczak M, Tomczak E. The need to report effect size estimates revisited. An overview of some recommended measures of effect size. Trends Sport Sci 2014; 1: 19-25
• 27 Falchikov N, Boud D. Student self-assessment in higher education: a meta-analysis. Rev Ed Res 1989;
• 28 Sargeant J, Armson H, Chesluk B. et al. The processes and dimensions of informed self-assessment: a conceptual model. Acad Med 2010; 85: 1212-1220
• 29 Scaffidi MA, Grover SC, Carnahan H. et al. Impact of experience on self-assessment accuracy of clinical colonoscopy competence. Gastrointest Endosc 2018; 87: 827-836
• 30 Vnuk A, Owen H, Plummer J. Assessing proficiency in adult basic life support: student and expert assessment and the impact of video recording. Med Teach 2006; 28: 429-434
• 31 Hawkins SC, Osborne A, Schofield SJ. et al. Improving the accuracy of self-assessment of practical clinical skills using video feedback--the importance of including benchmarks. Med Teach 2012; 34: 279-284
• 32 Kruger J, Dunning D. Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Pers Soc Psychol 1999; 77: 1121-1134
• 33 Khan R, Scaffidi MA, Walsh CM. et al. Simulation-based training of non-technical skills in colonoscopy: protocol for a randomized controlled trial. JMIR Res Protoc 2017; 6: e153
• 34 Scaffidi MA, Grover SC, Carnahan H. et al. A prospective comparison of live and video-based assessments of colonoscopy performance. Gastrointest Endosc 2017; 87: 688-694.e2
• 35 Eva KW, Armson H, Holmboe E. et al. Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes. Adv Health Sci Educ Theory Pract 2012; 171 5-26
• 36 Eva KW, Munoz J, Hanson MD. et al. Which factors, personal or external, most influence students’ generation of learning goals?. Acad Med 2010; 85: S102-S105