Validity evidence for simulator-based obstetric ultrasound competency assessment tool: a multi-center study

Abstract

Purpose To collect validity evidence for a simulator-based obstetric ultrasound competency assessment tool (OUCAT).

Methods 89 sonographers from three centers (XY, MC, DT), including novices (n=21), experienced trainees (n=44), and experts (n=24), participated in the competency assessment. Validity evidence of OUCAT was collected according to Standards for Educational and Psychological Testing. Content validity was ensured by reviewing guidelines and reaching expert consensus. The response process was ensured via training raters. Internal structure was explored through internal consistency, inter-rater reliability, and test-retest reliability. Relations to other variables were explored by comparing OUCAT scores of sonographers with different experience. Evidence for consequences was collected by determining the pass/fail level.

Results OUCAT included 123 items, of which 117 items could effectively distinguish novices from experts (P<0.05). The internal consistency was represented by the Cronbach’s α coefficient (0.978). The inter-rater reliability was high, with XY being 0.868, MC being 0.877, and DT being 0.937 (P<0.001). Test-retest reliability was 0.732 (P=0.001). The performance of experts was significantly better than experienced trainees, and the performance of experienced trainees was significantly better than novices (70.3±10.7 vs. 39.8±15.0 vs. 20.5±10.6, P<0.001). The pass/fail level determined by contrast group method was 45 points. The passing rate of novices, experienced trainees and experts was 0% (0/21), 31.8% (14/44), and 100% (24/24), respectively.

Conclusion Simulator-based OUCAT exhibits good reliability and validity in assessing obstetric ultrasound skills.

Zusammenfassung

Ziel Erhebung von Validitätsnachweisen für ein simulatorbasiertes Tool zur Beurteilung der Ultraschallkompetenz in der Geburtshilfe (OUCAT).

Material und Methoden 89 Sonografen aus 3 Zentren (XY, MC, DT), darunter Anfänger (n=21), erfahrene Auszubildende (n=44) und Experten (n=24), nahmen an der Kompetenzbewertung teil. Der Validitätsnachweis für OUCAT wurde gemäß den Standards für pädagogische und psychologische Tests ermittelt. Die inhaltliche Validität wurde durch die Überprüfung von Richtlinien und die Erzielung eines Expertenkonsenses sichergestellt. Der Antwortprozess wurde durch die Schulung von Bewertern sichergestellt. Die interne Struktur wurde durch interne Konsistenz, Inter-Rater-Reliabilität und Test-Retest-Reliabilität untersucht. Die Relationen zu anderen Variablen wurden durch den Vergleich der OUCAT-Scores der Sonografen mit der unterschiedlichen Erfahrung untersucht. Der Nachweis von Konsequenzen wurde erfasst, indem das Pass/Fail-Level bestimmt wurde.

Ergebnisse OUCAT umfasste 123 Elemente, von denen 117 effektiv zwischen Anfängern und Experten unterscheiden konnten (P<0,05). Die interne Konsistenz wurde durch den Cronbachs-α-Koeffizienten (0,978) dargestellt. Die Inter-Rater-Reliabilität war hoch: XY lag bei 0,868, MC bei 0,877 und DT bei 0,937 (p<0,001). Die Test-Retest-Reliabilität betrug 0,732 (p=0,001). Die Leistung von Experten war signifikant besser als die von erfahrenen Auszubildenden, und die Leistung von erfahrenen Auszubildenden war signifikant besser als die von Anfängern (70,3 ±10,7 vs. 39,8 ±15,0 vs. 20,5 ±10,6; p<0,001). Das durch die Kontrastgruppenmethode ermittelte Pass/Fail-Level betrug 45 Punkte. Die Quote für Bestehen betrug bei Anfängern 0% (0/21), bei erfahrenen Auszubildenden 31,8% (14/44) und bei Experten 100% (24/24).

Schlussfolgerung Das simulatorbasierte OUCAT zeigt eine gute Reliabilität und Validität bei der Bewertung von Ultraschallfähigkeiten in der Geburtshilfe.

Keywords

obstetric ultrasound - competency - assessment - simulation - validity

Publication History

Received: 07 October 2022

Accepted after revision: 04 July 2023

Accepted Manuscript online:
04 July 2023

Article published online:
06 October 2023

• References

• 1 Salomon LJ, Alfirevic Z, Berghella V. et al. Practice guidelines for performance of the routine mid-trimester fetal ultrasound scan. Ultrasound Obstet Gynecol 2011; 37: 116-126
• 2 Chalouhi GE, Bernardi V, Ville Y. Ultrasound simulators in obstetrics and gynecology: state of the art. Ultrasound Obstet Gynecol 2015; 46: 255-263
• 3 ISUOG Education Committee recommendations for basic training in obstetric and gynecological ultrasound. Ultrasound Obstet Gynecol 2014; 43: 113-116
• 4 Damewood SC, Leo M, Bailitz J. et al. Tools for Measuring Clinical Ultrasound Competency: Recommendations From the Ultrasound Competency Work Group. AEM Educ Train 2020; 4: S106-S112
• 5 Lockyer J, Carraccio C, Chan MK. et al. Core principles of assessment in competency-based medical education. Med Teach 2017; 39: 609-616
• 6 Abuhamad A, Minton KK, Benson CB. et al. Obstetric and gynecologic ultrasound curriculum and competency assessment in residency training programs: consensus report. Ultrasound Obstet Gynecol 2018; 51: 150-155
• 7 Gill RW. The Physics and Technology of Diagnostic Ultrasound: A Practitioner’s Guide. Sydney: High Frequency Publishing; 2012
• 8 Taksoe-Vester C, Dyre L, Schroll J. et al. Simulation-Based Ultrasound Training in Obstetrics and Gynecology: A Systematic Review and Meta-Analysis. Ultraschall in Med 2021; 42: e42-e54
• 9 Le Lous M, Klein M, Tesson C. et al. Metrics used to evaluate obstetric ultrasound skills on simulators: A systematic review. Eur J Obstet Gynecol Reprod Biol 2021; 258: 16-22
• 10 Tolsgaard MG, Ringsted C, Dreisler E. et al. Reliable and valid assessment of ultrasound operator competence in obstetrics and gynecology. Ultrasound Obstet Gynecol 2014; 43: 437-443
• 11 Tolsgaard MG, Ringsted C, Dreisler E. et al. Sustained effect of simulation-based ultrasound training on clinical performance: a randomized trial. Ultrasound Obstet Gynecol 2015; 46: 312-318
• 12 Chalouhi GE, Bernardi V, Gueneuc A. et al. Evaluation of trainees’ ability to perform obstetrical ultrasound using simulation: challenges and opportunities. Am J Obstet Gynecol 2016; 214: 525.e1-525.e8
• 13 American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 2014
• 14 Ostergaard ML, Konge L, Kahr N. et al. Four Virtual-Reality Simulators for Diagnostic Abdominal Ultrasound Training in Radiology. Diagnostics (Basel) 2019; 9: 50
• 15 Le Lous M, Tsatsaris V, Tesniere A. et al. Improving students’ ability to perform a standardized foetal biometry plane using ultrasound simulators. J Gynecol Obstet Hum Reprod 2017; 46: 439-443
• 16 Dyre L, Norgaard LN, Tabor A. et al. Collecting Validity Evidence for the Assessment of Mastery Learning in Simulation-Based Ultrasound Training. Ultraschall in Med 2016; 37: 386-392
• 17 Dromey BP, Ahmed S, Vasconcelos F. et al. Dimensionless squared jerk: An objective differential to assess experienced and novice probe movement in obstetric ultrasound. Prenat Diagn 2021; 41: 271-277
• 18 Rosen H, Windrim R, Lee YM. et al. Simulator Based Obstetric Ultrasound Training: A Prospective, Randomized Single-Blinded Study. J Obstet Gynaecol Can 2017; 39: 166-173
• 19 Burden C, Preshaw J, White P. et al. Usability of virtual-reality simulation training in obstetric ultrasonography: a prospective cohort study. Ultrasound Obstet Gynecol 2013; 42: 213-217
• 20 Madsen ME, Norgaard LN, Tabor A. et al. The Predictive Value of Ultrasound Learning Curves Across Simulated and Clinical Settings. J Ultrasound Med 2017; 36: 201-208
• 21 Salomon LJ, Winer N, Bernard JP. et al. A score-based method for quality control of fetal images at routine second-trimester ultrasound examination. Prenat Diagn 2008; 28: 822-827
• 22 Thomas PA, Kern DE, Hughes MT. et al. Curriculum development for medical education: a six-step approach. 4th edition. Baltimore: Johns Hopkins University Press; 2022