Validation of the second-generation Olympus colonoscopy simulator for skills assessment

Background and study aims: Simulators have potential value in providing objective evidence of technical skill for procedures within medicine. The aim of this study was to determine face and construct validity for the Olympus colonoscopy simulator and to establish which assessment measures map to clinical benchmarks of expertise.

Patients and methods: Thirty-four participants were recruited: 10 novices with no prior colonoscopy experience, 13 intermediate (trainee) endoscopists with fewer than 1000 previous colonoscopies, and 11 experienced endoscopists with more than 1000 previous colonoscopies. All participants completed three standardized cases on the simulator and experts gave feedback regarding the realism of the simulator. Forty metrics recorded automatically by the simulator were analyzed for their ability to distinguish between the groups.

Results: The simulator discriminated participants by experience level for 22 different parameters. Completion rates were lower for novices than for trainees and experts (37 % vs. 79 % and 88 % respectively, P < 0.001) and both novices and trainees took significantly longer to reach all major landmarks than the experts. Several technical aspects of competency were discriminatory; pushing with an embedded tip (P = 0.03), correct use of the variable stiffness function (P = 0.004), number of sigmoid N-loops (P = 0.02); size of sigmoid N-loops (P = 0.01), and time to remove alpha loops (P = 0.004). Out of 10, experts rated the realism of movement at 6.4, force feedback at 6.6, looping at 6.6, and loop resolution at 6.8.

Conclusions: The Olympus colonoscopy simulator has good face validity and excellent construct validity. It provides an objective assessment of colonoscopic skill on multiple measures and benchmarks have been set to allow its use as both a formative and a summative assessment tool.

References

• 1 Marshall J B. Technical proficiency of trainees performing colonoscopy: a learning curve.  Gastrointest Endosc. 1995;  42 287-291
• 2 Wigton R S, Nicolas J A, Blank L L. Procedural skills of the general internist. A survey of 2500 physicians.  Ann Intern Med. 1989;  111 1023-1034
• 3 Tassios P S, Ladas S D, Grammenos I. et al . Acquisition of competence in colonoscopy: the learning curve of trainees.  Endoscopy. 1999;  31 702-706
• 4 Wexner S D, Eisen G M, Simmang C. Principles of privileging and credentialing for endoscopy and colonoscopy.  Surg Endosc. 2002;  16 367-369
• 5 Cass O, Freeman M L, Cohen J. et al . Acquisition of competency in endoscopic skills (ACES) during training; a multicenter study.  Gastrointest Endosc. 1996;  43 308
• 6 Cass O W. Objective evaluation of competence: technical skills in gastrointestinal endoscopy.  Endoscopy. 1995;  27 86-89
• 7 Parry B R, Williams S M. Competency and the colonoscopist: a learning curve.  Aust N Z J Surg. 1991;  61 419-422
• 8 Church J, Oakley J, Milsom J. et al . Colonoscopy training: the need for patience (patients).  Aust N Z J Surg. 2002;  72 89-91
• 9 Bowles C J, Leicester R, Romaya C. et al . A prospective study of colonoscopy practice in the UK today: are we adequately prepared for national colorectal cancer screening tomorrow?.  Gut. 2004;  53 277-283
• 10 Sarker S K, Albrani T, Zaman A, Patel B. Procedural performance in gastrointestinal endoscopy: an assessment and self-appraisal tool.  Am J Surg. 2008;  196 450-455
• 11 Park J, MacRae H, Musselman L J. et al . Randomized controlled trial of virtual reality simulator training: transfer to live patients.  Am J Surg. 2007;  194 205-211
• 12 Grantcharov T P, Carstensen L, Schulze S. Objective assessment of gastrointestinal endoscopy skills using a virtual reality simulator.  JSLS. 2005;  9 130-133
• 13 Felsher J J, Olesevich M, Farres H. et al . Validation of a flexible endoscopy simulator.  Am J Surg. 2005;  189 497-500
• 14 Sedlack R E, Kolars J C. Validation of a computer-based colonoscopy simulator.  Gastrointest Endosc. 2003;  57 214-218
• 15 Mahmood T, Darzi A. A study to validate the colonoscopy simulator.  Surg Endosc. 2003;  17 1583-1589
• 16 MacDonald J, Ketchum J, Williams R G, Rogers L Q. A lay person versus a trained endoscopist: can the PreOp endoscopy simulator detect a difference?.  Surg Endosc. 2003;  17 896-898
• 17 Datta V, Mandalia M, Mackay S, Darzi A. The PreOp flexible sigmoidoscopy trainer. Validation and early evaluation of a virtual reality based system.  Surg Endosc. 2002;  16 1459-1463
• 18 Moorthy K, Munz Y, Orchard T R. et al . An innovative method for the assessment of skills in lower gastrointestinal endoscopy.  Surg Endosc. 2004;  18 1613-1619
• 19 Koch A D, Buzink S N, Heemskerk J. et al . Expert and construct validity of the Simbionix GI Mentor II endoscopy simulator for colonoscopy.  Surg Endosc. 2007;  22 158-162
• 20 Mahmood T, Darzi A. The incidence of bowel perforations is positively related to the intensity of pain experienced during simulated colonoscopy.  Gastrointest Endosc. 2004;  59 P125, S1554
• 21 Aabakken L, Adamsen S, Kruse A. Performance of a colonoscopy simulator: experience from a hands-on endoscopy course.  Endoscopy. 2000;  32 911-913
• 22 Fitzgerald T N, Duffy A J, Bell R L. et al . Computer-based endoscopy simulation: emerging roles in teaching and professional skills assessment.  J Surg Educ. 2008;  65 229-235
• 23 Gerson L B. Evidence-based assessment of endoscopic simulators for training.  Gastrointest Endosc Clin N Am. 2006;  16 489-509, vii-viii
• 24 Williams C B, Thomas-Gibson S. Rational colonoscopy, realistic simulation, and accelerated teaching.  Gastrointest Endosc Clin N Am. 2006;  16 457-470
• 25 Koch A D, Haringsma J, Schoon E J. et al . A second-generation virtual reality simulator for colonoscopy: validation and initial experience.  Endoscopy. 2008;  40 735-738
• 26 Shah S G, Saunders B P, Brooker J C, Williams C B. Magnetic imaging of colonoscopy: an audit of looping, accuracy and ancillary maneuvers.  Gastrointest Endosc. 2000;  52 1-8
• 27 NHS Modernisation Agency .Improving endoscopy services. Meeting the challenges. Leicester, UK; NHS Modernisation Agency 2004
• 28 General Medical Council .Principles of good medical education and training. 2004
• 29 Joint Advisory Group for Gastrointestinal Endoscopy .Guidelines for the training, appraisal and assessment of trainees in gastrointestinal endoscopy. British Society of Gastroenterology, London; 2004. Available from: http://www.bsg.org.uk/pdf_word_docs/jag_recommendations_2004.pdf
• 30 Michelson J D, Manning L. Competency assessment in simulation-based procedural education.  Am J Surg. 2008;  196 609-615
• 31 Sedlack R E, Baron T H, Downing S M, Schwartz A J. Validation of a colonoscopy simulation model for skills assessment.  Am J Gastroenterol. 2007;  102 64-74
• 32 Shah S G, Brooker J C, Williams C B. et al . Effect of magnetic endoscope imaging on colonoscopy performance: a randomised controlled trial.  Lancet. 2000;  356 1718-1722
• 33 Hoff G, Bretthauer M, Dahler S. et al . Improvement in caecal intubation rate and pain reduction by using 3-dimensional magnetic imaging for unsedated colonoscopy: a randomized trial of patients referred for colonoscopy.  Scand J Gastroenterol. 2007;  42 885-889
• 34 Wehrmann K, Fruhmorgen P. Evaluation of a new three-dimensional magnetic imaging system for use during colonoscopy.  Endoscopy. 2002;  34 905-908
• 35 Cheung H Y, Chung C C, Kwok S Y. et al . Improvement in colonoscopy performance with adjunctive magnetic endoscope imaging: a randomized controlled trial.  Endoscopy. 2006;  38 214-217
• 36 Shah S G, Pearson H J, Moss S. et al . Magnetic endoscope imaging: a new technique for localizing colonic lesions.  Endoscopy. 2002;  34 900-904
• 37 Shah S G, Brooker J C, Thapar C. et al . Effect of magnetic endoscope imaging on patient tolerance and sedation requirements during colonoscopy: a randomized controlled trial.  Gastrointest Endosc. 2002;  55 832-837