Correlation of Ophthalmology Residency Application Characteristics with Subsequent Performance in Residency

Abstract

Purpose Only from reviewing applications, it is difficult to identify which applicants will be successful ophthalmology residents. The change of the USMLE Step 1 scoring to “Pass/Fail” removes another quantitative metric. We aimed to identify application attributes correlated with successful residency performance. This study also used artificial intelligence (AI) to evaluate letters of recommendation (LOR), the Dean's letter (MSPE), and personal statement (PS).

Design Retrospective analysis of application characteristics versus residency performance was conducted.

Participants Residents who graduated from the Dean McGee Eye Institute/University of Oklahoma Ophthalmology residency from 2004 to 2019 were included in this study.

Methods Thirty-four attributes were recorded from each application. Residents were subjectively ranked into tertiles and top and bottom deciles based on residency performance by faculty present during their training. The Ophthalmic Knowledge Assessment Program (OKAP) examination scores were used as an objective performance metric. Analysis was performed to identify associations between application attributes and tertile/decile ranking. Additional analysis used AI and natural language processing to evaluate applicant LORs, MSPE, and PS.

Main Outcome Measures Characteristics from residency applications that correlate with resident performance were the primary outcome of this study.

Results Fifty-five residents and 21 faculty members were included. A grade of “A” or “Honors” in the obstetrics/gynecology (OB/GYN) clerkship and the presence of a home ophthalmology department were associated with ranking in the top tertile but not the top decile. Mean core clerkship grades, medical school ranking in the top 25 U.S. News and World Report (USNWR) primary care rankings, and postgraduate year (PGY)-2 and PGY-3 OKAP scores were predictive of being ranked in both the top tertile and the top decile. USMLE scores, alpha-omega-alpha (AOA) status, and number of publications did not correlate with subjective resident performance. AI analysis of LORs, MSPE, and PS did not identify any text features that correlated with resident performance.

Conclusions Many metrics traditionally felt to be predictive of residency success (USMLE scores, AOA status, and research) did not predict resident success in our study. We did confirm the importance of core clerkship grades and medical school ranking. Objective measures of success such as PGY-2 and PGY-3 OKAP scores were associated with high subjective ranking.

Keywords

Residency Application - resident performance - artificial intelligence

Meeting Presentation

This study was previously presented in part as a poster at the annual Association of University Professors of Ophthalmology (AUPO) meeting, 2019.

Conflict of Interest

No conflicting relationship exists for any author.

Publikationsverlauf

Eingereicht: 28. November 2020

Angenommen: 03. April 2021

Artikel online veröffentlicht:
10. November 2021

• References

• 1 Ophthalmology Match Summary Report 2019. Accessed May 7, 2020 at: https://www.sfmatch.org/PDFFilesDisplay/Ophthalmology_Residency_Stats_2019.pdf
• 2 National Resident Matching Program. Results and Data: 2018 main residency match. Accessed May 7, 2020 at: http://www.nrmp.org/wp-content/uploads/2018/04/Main-Match-Result-and-Data-2018.pdf
• 3 Nallasamy S, Uhler T, Nallasamy N, Tapino PJ, Volpe NJ. Ophthalmology resident selection: current trends in selection criteria and improving the process. Ophthalmology 2010; 117 (05) 1041-1047
• 4 Siatkowski RM, Mian SI, Culican SM. et al; Association of University Professors of Ophthalmology. Probability of success in the ophthalmology residency match: three-year outcomes analysis of San Francisco matching program data. J Acad Ophthalmol 2018; 10 (01) e150-e157
• 5 United States Medical Licensing Examination. Change to pass/fail score reporting for step 1. Accessed May 7, 2020 at: https://www.usmle.org/incus/
• 6 Thompson RH, Lohse CM, Husmann DA, Leibovich BC, Gettman MT. Predictors of a successful urology resident using medical student application materials. Urology 2017; 108: 22-28
• 7 Tolan AM, Kaji AH, Quach C, Hines OJ, de Virgilio C. The electronic residency application service application can predict accreditation council for graduate medical education competency-based surgical resident performance. J Surg Educ 2010; 67 (06) 444-448
• 8 Brothers TE, Wetherholt S. Importance of the faculty interview during the resident application process. J Surg Educ 2007; 64 (06) 378-385
• 9 Raman T, Alrabaa RG, Sood A, Maloof P, Benevenia J, Berberian W. Does residency selection criteria predict performance in orthopaedic surgery residency?. Clin Orthop Relat Res 2016; 474 (04) 908-914
• 10 Boyse TD, Patterson SK, Cohan RH. et al. Does medical school performance predict radiology resident performance?. Acad Radiol 2002; 9 (04) 437-445
• 11 Erlandson EE, Calhoun JG, Barrack FM. et al. Resident selection: applicant selection criteria compared with performance. Surgery 1982; 92 (02) 270-275
• 12 Fine PL, Hayward RA. Do the criteria of resident selection committees predict residents' performances?. Acad Med 1995; 70 (09) 834-838
• 13 Khongphatthanayothin A, Chongsrisawat V, Wananukul S, Sanpavat S. Resident recruitment: what are good predictors for performance during pediatric residency training?. J Med Assoc Thai 2002; 85 (Suppl. 01) S302-S311
• 14 Dirschl DR, Dahners LE, Adams GL, Crouch JH, Wilson FC. Correlating selection criteria with subsequent performance as residents. Clin Orthop Relat Res 2002; (399) 265-271
• 15 Egol KA, Collins J, Zuckerman JD. Success in orthopaedic training: resident selection and predictors of quality performance. J Am Acad Orthop Surg 2011; 19 (02) 72-80
• 16 Harfmann KL, Zirwas MJ. Can performance in medical school predict performance in residency? A compilation and review of correlative studies. J Am Acad Dermatol 2011; 65 (05) 1010-1022.e2
• 17 Spitzer AB, Gage MJ, Looze CA, Walsh M, Zuckerman JD, Egol KA. Factors associated with successful performance in an orthopaedic surgery residency. J Bone Joint Surg Am 2009; 91 (11) 2750-2755
• 18 Chole RA, Ogden MA. Predictors of future success in otolaryngology residency applicants. Arch Otolaryngol Head Neck Surg 2012; 138 (08) 707-712
• 19 Grewal SG, Yeung LS, Brandes SB. Predictors of success in a urology residency program. J Surg Educ 2013; 70 (01) 138-143
• 20 Hayden SR, Hayden M, Gamst A. What characteristics of applicants to emergency medicine residency programs predict future success as an emergency medicine resident?. Acad Emerg Med 2005; 12 (03) 206-210
• 21 Burish MJ, Fredericks CA, Engstrom JW, Tateo VL, Josephson SA. Predicting success: What medical student measures predict resident performance in neurology?. Clin Neurol Neurosurg 2015; 135: 69-72
• 22 Metro DG, Talarico JF, Patel RM, Wetmore AL. The resident application process and its correlation to future performance as a resident. Anesth Analg 2005; 100 (02) 502-505
• 23 Chen F, Arora H, Martinelli SM. et al. The predictive value of pre-recruitment achievement on resident performance in anesthesiology. J Clin Anesth 2017; 39: 139-144
• 24 Stohl HE, Hueppchen NA, Bienstock JL. Can medical school performance predict residency performance? Resident selection and predictors of successful performance in obstetrics and gynecology. J Grad Med Educ 2010; 2 (03) 322-326
• 25 Alterman DM, Jones TM, Heidel RE, Daley BJ, Goldman MH. The predictive value of general surgery application data for future resident performance. J Surg Educ 2011; 68 (06) 513-518
• 26 Bowe SN, Laury AM, Gray ST. Associations between otolaryngology applicant characteristics and future performance in residency or practice: a systematic review. Otolaryngol Head Neck Surg 2017; 156 (06) 1011-1017
• 27 Daly KA, Levine SC, Adams GL. Predictors for resident success in otolaryngology. J Am Coll Surg 2006; 202 (04) 649-654
• 28 Borowitz SM, Saulsbury FT, Wilson WG. Information collected during the residency match process does not predict clinical performance. Arch Pediatr Adolesc Med 2000; 154 (03) 256-260
• 29 Black KP, Abzug JM, Chinchilli VM. Orthopaedic in-training examination scores: a correlation with USMLE results. J Bone Joint Surg Am 2006; 88 (03) 671-676
• 30 Carmichael KD, Westmoreland JB, Thomas JA, Patterson RM. Relation of residency selection factors to subsequent orthopaedic in-training examination performance. South Med J 2005; 98 (05) 528-532
• 31 Gunderman RB, Jackson VP. Are NBME examination scores useful in selecting radiology residency candidates?. Acad Radiol 2000; 7 (08) 603-606
• 32 Chen B, Zaebst D, Seel L. A macro to calculate kappa statistics for categorizations by multiple raters. Accessed July 23, 2021 at: https://support.sas.com/resources/papers/proceedings/proceedings/sugi30/155-30.pdf
• 33 Hu M, Liu B. Mining and summarizing customer reviews. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD-2004), Seattle, Washington: 2004: 168-177
• 34 Nielsen FA. A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. Proceedings of the ESWC2011 Workshop on ‘Making Sense of Microposts’: Big things come in small packages 2011: 93-98
• 35 Loughran T, McDonald B. When is a liability not a liability? Textual analysis, dictionaries, and 10-Ks. J Finance 2011; 66 (01) 35-65
• 36 Mohammad S, Turney P. Crowdsourcing a word-emotion association lexicon. Comput Intell 2013; 29 (03) 436-465
• 37 Lin F, Oh SK, Gordon LK, Pineles SL, Rosenberg JB, Tsui I. Gender-based differences in letters of recommendation written for ophthalmology residency applicants. BMC Med Educ 2019; 19 (01) 476
• 38 Grova MM, Jenkins FG, Filippou P. et al. Gender bias in surgical oncology fellowship recommendation letters: gaining progress. J Surg Educ 2021; 78 (03) 866-874
• 39 Kobayashi AN, Sterling RS, Tackett SA, Chee BW, Laporte DM, Humbyrd CJ. Are there gender-based differences in language in letters of recommendation to an orthopaedic surgery residency program?. Clin Orthop Relat Res 2020; 478 (07) 1400-1408
• 40 Madera JM, Hebl MR, Martin RC. Gender and letters of recommendation for academia: agentic and communal differences. J Appl Psychol 2009; 94 (06) 1591-1599
• 41 Trix F, Psenka C. Exploring the color of glass: letters of recommendation for female and male medical faculty. Discourse Soc 2003; 14 (02) 191-220
• 42 Powers A, Gerull KM, Rothman R, Klein SA, Wright RW, Dy CJ. Race- and gender-based differences in descriptions of applicants in the letters of recommendation for orthopaedic surgery residency. JBJS Open Access 2020; 5 (03) e20 .00023-e20.00023
• 43 Filippou P, Mahajan S, Deal A. et al. The presence of gender bias in letters of recommendations written for urology residency applicants. Urology 2019; 134 (01) 56-61
• 44 Lee AG, Golnik KC, Oetting TA. et al. Re-engineering the resident applicant selection process in ophthalmology: a literature review and recommendations for improvement. Surv Ophthalmol 2008; 53 (02) 164-176
• 45 Takayama H, Grinsell R, Brock D, Foy H, Pellegrini C, Horvath K. Is it appropriate to use core clerkship grades in the selection of residents?. Curr Surg 2006; 63 (06) 391-396