Improving the Performance Metric of an Established Coronary Surgery Simulation Programme to Reward a Training Focus on Skill Quality and Proficiency

Background: We have previously developed a coronary surgery simulation program based on the “deliberate practice” approach (i.e., with the specific aim to improve performance). Various curriculum adjustments (e.g., length of training periods, on site vs. home training) have been made so far. We noticed that, with each curriculum adjustment toward a higher training intensity, trainees focused their training efforts more and more toward task completion time instead of skill quality. This study assessed the effect of an adjustment of the performance metric on overall performance.

Method: A multidimensional assessment matrix was used to evaluate video recordings: quality parameters—Likert scales 1 to 5 for handling of instruments, motion with the needle, and tissue handling. Handling and motion were doubly weighed. Completion time/stich was measured. The matrix was used in three student training protocols. Protocol 1: six weeks, home-based training; three on-site workshops (n = 19). Protocol 2: four days, on-site training, continuous feedback (net training time: 24 hours; N = 16). Protocol 3: four days mixed curriculum (12 hours on site, 12 hours home based; N = 16). Two groups of fellows and residents served as controls. Original performance metric: average skill score/completion time; target: 60/min. Recalibrated performance metric: skill score: maximum 30 point, quality bonus (average skill score >4.5): 10 point, completion time: maximum 20 point, target: 60 point.

Results: With the original performance metric, students seemingly outperformed fellows after a 24-hour hybrid (on site and home based) training curriculum despite a significantly lower median skill score (4.1 [3.7–4.4] vs. 4.8 [4.7–4.9], p = 0.0017) due to faster completion times (6.6 second/stich [5.8–8.6] vs. 11 second/stich [10–12]; p < 0.0001): Original performance metric: students versus fellows: 36 (27–43) versus 27 (24–29), p = 0.047. Recalibrated performance metric: 41 (36–43) versus 47 (45–48), p < 0.0001. Similar effects were seen in comparison with a group of residents.

Conclusion: Our study shows the importance of the design of the overall performance metric of surgical simulations to reward a training focus on skill quality rather than completion times. Recalibration resulted in a more clinically relevant appraisal of skill quality. Future training workshops will be performed using the new performance metric. Surgical simulation training including performance assessment based on the “deliberate practice” principle should be a mandatory part of surgical training.

Publication History

Article published online:
28 January 2023