Download PDF
CC BY-NC-ND 4.0 · Journal of Academic Ophthalmology 2021; 13(02): e151-e157
DOI: 10.1055/s-0041-1733932
Research Article

Correlation of Ophthalmology Residency Application Characteristics with Subsequent Performance in Residency

Brett M. Gudgel
1   University of Oklahoma Health Science Center, Oklahoma City, Oklahoma
,
Andrew T. Melson
1   University of Oklahoma Health Science Center, Oklahoma City, Oklahoma
,
Justin Dvorak
1   University of Oklahoma Health Science Center, Oklahoma City, Oklahoma
,
Kai Ding
1   University of Oklahoma Health Science Center, Oklahoma City, Oklahoma
,
R. Michael Siatkowski
1   University of Oklahoma Health Science Center, Oklahoma City, Oklahoma
› Author Affiliations
› Further Information
   Permissions and Reprints

Abstract

Purpose Only from reviewing applications, it is difficult to identify which applicants will be successful ophthalmology residents. The change of the USMLE Step 1 scoring to “Pass/Fail” removes another quantitative metric. We aimed to identify application attributes correlated with successful residency performance. This study also used artificial intelligence (AI) to evaluate letters of recommendation (LOR), the Dean's letter (MSPE), and personal statement (PS).

Design Retrospective analysis of application characteristics versus residency performance was conducted.

Participants Residents who graduated from the Dean McGee Eye Institute/University of Oklahoma Ophthalmology residency from 2004 to 2019 were included in this study.

Methods Thirty-four attributes were recorded from each application. Residents were subjectively ranked into tertiles and top and bottom deciles based on residency performance by faculty present during their training. The Ophthalmic Knowledge Assessment Program (OKAP) examination scores were used as an objective performance metric. Analysis was performed to identify associations between application attributes and tertile/decile ranking. Additional analysis used AI and natural language processing to evaluate applicant LORs, MSPE, and PS.

Main Outcome Measures Characteristics from residency applications that correlate with resident performance were the primary outcome of this study.

Results Fifty-five residents and 21 faculty members were included. A grade of “A” or “Honors” in the obstetrics/gynecology (OB/GYN) clerkship and the presence of a home ophthalmology department were associated with ranking in the top tertile but not the top decile. Mean core clerkship grades, medical school ranking in the top 25 U.S. News and World Report (USNWR) primary care rankings, and postgraduate year (PGY)-2 and PGY-3 OKAP scores were predictive of being ranked in both the top tertile and the top decile. USMLE scores, alpha-omega-alpha (AOA) status, and number of publications did not correlate with subjective resident performance. AI analysis of LORs, MSPE, and PS did not identify any text features that correlated with resident performance.

Conclusions Many metrics traditionally felt to be predictive of residency success (USMLE scores, AOA status, and research) did not predict resident success in our study. We did confirm the importance of core clerkship grades and medical school ranking. Objective measures of success such as PGY-2 and PGY-3 OKAP scores were associated with high subjective ranking.

Keywords

Residency Application - resident performance - artificial intelligence

Meeting Presentation

This study was previously presented in part as a poster at the annual Association of University Professors of Ophthalmology (AUPO) meeting, 2019.


Conflict of Interest

No conflicting relationship exists for any author.




Publication History

Received: 28 November 2020

Accepted: 03 April 2021

Article published online:
10 November 2021

© 2021. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)

Thieme Medical Publishers, Inc.
333 Seventh Avenue, 18th Floor, New York, NY 10001, USA

 

  • References

  • 1 Ophthalmology Match Summary Report 2019. Accessed May 7, 2020 at: https://www.sfmatch.org/PDFFilesDisplay/Ophthalmology_Residency_Stats_2019.pdf
    PubMed
  • 2 National Resident Matching Program. Results and Data: 2018 main residency match. Accessed May 7, 2020 at: http://www.nrmp.org/wp-content/uploads/2018/04/Main-Match-Result-and-Data-2018.pdf
    PubMed
  • 3 Nallasamy S, Uhler T, Nallasamy N, Tapino PJ, Volpe NJ. Ophthalmology resident selection: current trends in selection criteria and improving the process. Ophthalmology 2010; 117 (05) 1041-1047
    CrossrefPubMedSearch in Google Scholar
  • 4 Siatkowski RM, Mian SI, Culican SM. et al; Association of University Professors of Ophthalmology. Probability of success in the ophthalmology residency match: three-year outcomes analysis of San Francisco matching program data. J Acad Ophthalmol 2018; 10 (01) e150-e157
    Thieme ConnectPubMedSearch in Google Scholar
  • 5 United States Medical Licensing Examination. Change to pass/fail score reporting for step 1. Accessed May 7, 2020 at: https://www.usmle.org/incus/
    PubMed
  • 6 Thompson RH, Lohse CM, Husmann DA, Leibovich BC, Gettman MT. Predictors of a successful urology resident using medical student application materials. Urology 2017; 108: 22-28
    CrossrefPubMedSearch in Google Scholar
  • 7 Tolan AM, Kaji AH, Quach C, Hines OJ, de Virgilio C. The electronic residency application service application can predict accreditation council for graduate medical education competency-based surgical resident performance. J Surg Educ 2010; 67 (06) 444-448
    CrossrefPubMedSearch in Google Scholar
  • 8 Brothers TE, Wetherholt S. Importance of the faculty interview during the resident application process. J Surg Educ 2007; 64 (06) 378-385
    CrossrefPubMedSearch in Google Scholar
  • 9 Raman T, Alrabaa RG, Sood A, Maloof P, Benevenia J, Berberian W. Does residency selection criteria predict performance in orthopaedic surgery residency?. Clin Orthop Relat Res 2016; 474 (04) 908-914
    CrossrefPubMedSearch in Google Scholar
  • 10 Boyse TD, Patterson SK, Cohan RH. et al. Does medical school performance predict radiology resident performance?. Acad Radiol 2002; 9 (04) 437-445
    CrossrefPubMedSearch in Google Scholar
  • 11 Erlandson EE, Calhoun JG, Barrack FM. et al. Resident selection: applicant selection criteria compared with performance. Surgery 1982; 92 (02) 270-275
    PubMedSearch in Google Scholar
  • 12 Fine PL, Hayward RA. Do the criteria of resident selection committees predict residents' performances?. Acad Med 1995; 70 (09) 834-838
    PubMedSearch in Google Scholar
  • 13 Khongphatthanayothin A, Chongsrisawat V, Wananukul S, Sanpavat S. Resident recruitment: what are good predictors for performance during pediatric residency training?. J Med Assoc Thai 2002; 85 (Suppl. 01) S302-S311
    PubMedSearch in Google Scholar
  • 14 Dirschl DR, Dahners LE, Adams GL, Crouch JH, Wilson FC. Correlating selection criteria with subsequent performance as residents. Clin Orthop Relat Res 2002; (399) 265-271
    CrossrefPubMedSearch in Google Scholar
  • 15 Egol KA, Collins J, Zuckerman JD. Success in orthopaedic training: resident selection and predictors of quality performance. J Am Acad Orthop Surg 2011; 19 (02) 72-80
    CrossrefPubMedSearch in Google Scholar
  • 16 Harfmann KL, Zirwas MJ. Can performance in medical school predict performance in residency? A compilation and review of correlative studies. J Am Acad Dermatol 2011; 65 (05) 1010-1022.e2
    CrossrefPubMedSearch in Google Scholar
  • 17 Spitzer AB, Gage MJ, Looze CA, Walsh M, Zuckerman JD, Egol KA. Factors associated with successful performance in an orthopaedic surgery residency. J Bone Joint Surg Am 2009; 91 (11) 2750-2755
    CrossrefPubMedSearch in Google Scholar
  • 18 Chole RA, Ogden MA. Predictors of future success in otolaryngology residency applicants. Arch Otolaryngol Head Neck Surg 2012; 138 (08) 707-712
    CrossrefPubMedSearch in Google Scholar
  • 19 Grewal SG, Yeung LS, Brandes SB. Predictors of success in a urology residency program. J Surg Educ 2013; 70 (01) 138-143
    CrossrefPubMedSearch in Google Scholar
  • 20 Hayden SR, Hayden M, Gamst A. What characteristics of applicants to emergency medicine residency programs predict future success as an emergency medicine resident?. Acad Emerg Med 2005; 12 (03) 206-210
    CrossrefPubMedSearch in Google Scholar
  • 21 Burish MJ, Fredericks CA, Engstrom JW, Tateo VL, Josephson SA. Predicting success: What medical student measures predict resident performance in neurology?. Clin Neurol Neurosurg 2015; 135: 69-72
    CrossrefPubMedSearch in Google Scholar
  • 22 Metro DG, Talarico JF, Patel RM, Wetmore AL. The resident application process and its correlation to future performance as a resident. Anesth Analg 2005; 100 (02) 502-505
    CrossrefPubMedSearch in Google Scholar
  • 23 Chen F, Arora H, Martinelli SM. et al. The predictive value of pre-recruitment achievement on resident performance in anesthesiology. J Clin Anesth 2017; 39: 139-144
    CrossrefPubMedSearch in Google Scholar
  • 24 Stohl HE, Hueppchen NA, Bienstock JL. Can medical school performance predict residency performance? Resident selection and predictors of successful performance in obstetrics and gynecology. J Grad Med Educ 2010; 2 (03) 322-326
    CrossrefPubMedSearch in Google Scholar
  • 25 Alterman DM, Jones TM, Heidel RE, Daley BJ, Goldman MH. The predictive value of general surgery application data for future resident performance. J Surg Educ 2011; 68 (06) 513-518
    CrossrefPubMedSearch in Google Scholar
  • 26 Bowe SN, Laury AM, Gray ST. Associations between otolaryngology applicant characteristics and future performance in residency or practice: a systematic review. Otolaryngol Head Neck Surg 2017; 156 (06) 1011-1017
    CrossrefPubMedSearch in Google Scholar
  • 27 Daly KA, Levine SC, Adams GL. Predictors for resident success in otolaryngology. J Am Coll Surg 2006; 202 (04) 649-654
    CrossrefPubMedSearch in Google Scholar
  • 28 Borowitz SM, Saulsbury FT, Wilson WG. Information collected during the residency match process does not predict clinical performance. Arch Pediatr Adolesc Med 2000; 154 (03) 256-260
    CrossrefPubMedSearch in Google Scholar
  • 29 Black KP, Abzug JM, Chinchilli VM. Orthopaedic in-training examination scores: a correlation with USMLE results. J Bone Joint Surg Am 2006; 88 (03) 671-676
    PubMedSearch in Google Scholar
  • 30 Carmichael KD, Westmoreland JB, Thomas JA, Patterson RM. Relation of residency selection factors to subsequent orthopaedic in-training examination performance. South Med J 2005; 98 (05) 528-532
    CrossrefPubMedSearch in Google Scholar
  • 31 Gunderman RB, Jackson VP. Are NBME examination scores useful in selecting radiology residency candidates?. Acad Radiol 2000; 7 (08) 603-606
    CrossrefPubMedSearch in Google Scholar
  • 32 Chen B, Zaebst D, Seel L. A macro to calculate kappa statistics for categorizations by multiple raters. Accessed July 23, 2021 at: https://support.sas.com/resources/papers/proceedings/proceedings/sugi30/155-30.pdf
    PubMed
  • 33 Hu M, Liu B. Mining and summarizing customer reviews. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD-2004), Seattle, Washington: 2004: 168-177
    PubMedSearch in Google Scholar
  • 34 Nielsen FA. A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. Proceedings of the ESWC2011 Workshop on ‘Making Sense of Microposts’: Big things come in small packages 2011: 93-98
    PubMed
  • 35 Loughran T, McDonald B. When is a liability not a liability? Textual analysis, dictionaries, and 10-Ks. J Finance 2011; 66 (01) 35-65
    CrossrefPubMedSearch in Google Scholar
  • 36 Mohammad S, Turney P. Crowdsourcing a word-emotion association lexicon. Comput Intell 2013; 29 (03) 436-465
    CrossrefPubMedSearch in Google Scholar
  • 37 Lin F, Oh SK, Gordon LK, Pineles SL, Rosenberg JB, Tsui I. Gender-based differences in letters of recommendation written for ophthalmology residency applicants. BMC Med Educ 2019; 19 (01) 476
    CrossrefPubMedSearch in Google Scholar
  • 38 Grova MM, Jenkins FG, Filippou P. et al. Gender bias in surgical oncology fellowship recommendation letters: gaining progress. J Surg Educ 2021; 78 (03) 866-874
    CrossrefPubMedSearch in Google Scholar
  • 39 Kobayashi AN, Sterling RS, Tackett SA, Chee BW, Laporte DM, Humbyrd CJ. Are there gender-based differences in language in letters of recommendation to an orthopaedic surgery residency program?. Clin Orthop Relat Res 2020; 478 (07) 1400-1408
    CrossrefPubMedSearch in Google Scholar
  • 40 Madera JM, Hebl MR, Martin RC. Gender and letters of recommendation for academia: agentic and communal differences. J Appl Psychol 2009; 94 (06) 1591-1599
    CrossrefPubMedSearch in Google Scholar
  • 41 Trix F, Psenka C. Exploring the color of glass: letters of recommendation for female and male medical faculty. Discourse Soc 2003; 14 (02) 191-220
    CrossrefPubMedSearch in Google Scholar
  • 42 Powers A, Gerull KM, Rothman R, Klein SA, Wright RW, Dy CJ. Race- and gender-based differences in descriptions of applicants in the letters of recommendation for orthopaedic surgery residency. JBJS Open Access 2020; 5 (03) e20 .00023-e20.00023
    CrossrefPubMedSearch in Google Scholar
  • 43 Filippou P, Mahajan S, Deal A. et al. The presence of gender bias in letters of recommendations written for urology residency applicants. Urology 2019; 134 (01) 56-61
    CrossrefPubMedSearch in Google Scholar
  • 44 Lee AG, Golnik KC, Oetting TA. et al. Re-engineering the resident applicant selection process in ophthalmology: a literature review and recommendations for improvement. Surv Ophthalmol 2008; 53 (02) 164-176
    CrossrefPubMedSearch in Google Scholar
  • 45 Takayama H, Grinsell R, Brock D, Foy H, Pellegrini C, Horvath K. Is it appropriate to use core clerkship grades in the selection of residents?. Curr Surg 2006; 63 (06) 391-396
    CrossrefPubMedSearch in Google Scholar