Keywords
ophthalmology residency - medical school - research - bibliometrics - h-index - residency match
Many factors influence an applicant's matriculation into an ophthalmology residency program. Previous studies have assessed the importance of quantifiable aspects of the application, such as board scores, Alpha Omega Alpha (AOA) status, and clinical grades in matching into ophthalmology.[1]
[2] Although research was mentioned in several of these studies, its effects on the match process has never been quantified. A survey of program directors, chairpersons, and people involved in matching committees indicated that a year of research was noted to be helpful in improving ophthalmology applications.[3]
In this study, we analyze the correlation between different research parameters and the rank of residency program matched, as measured by research output and reputation. Similar bibliometric studies have been done to evaluate the impact of research on matching into neurosurgery.[4] An objective analysis of the often-elusive term “research” can help medical students and advisors understand the impact of research on their residency application. We aim to create an accurate research profile of ophthalmology applicants by analyzing the papers published by current ophthalmology residents while they were applying to residency. An inherent limitation to our approach of primarily relying on the number of total publications is that it fails to fully capture the research contribution of medical students as listed on their San Francisco Match residency applications, which also include abstracts, presentations, and manuscripts in submission. However, access to this information is not readily available, and by focusing on the quality and quantity of published manuscripts we hope to provide a more uniform, objective analysis as to how research affects the chances of matching into ophthalmology.
Methods
Setting and Study Population
First, 117 residency programs using the public Accreditation Council for Graduate Medical Education (ACGME) database were identified (https://apps.acgme.org/ads/public/reports/report/1 with “Ophthalmology” as the specialty. Accessed September 24, 2017). Next, we obtained the names of the members of the Ophthalmology Class of 2019 from individual program Web sites. If this information was not verified, it was omitted from the analysis. Thirty programs which did not list their residents were omitted from the data analysis. From the Web sites, we noted each resident's name, medical school, and degree type.
After compiling a list of the applicants, we used the “Author Search” option on Scopus to look up each applicant individually. When an applicant's name did not yield any results, it was counted as a 0 for published papers and Hirsch's index (h-index) in the analysis. If results were unclear for an applicant, they were omitted from analysis. In total, eight applicants were removed for missing information, resulting in a final cohort of 340 applicants. For each profile, the number of published papers accepted on or prior to September 2014 were noted, to best reflect the number of fully published manuscripts each applicant reported to the San Francisco match.
For each applicant search on Scopus, we noted the number of papers published within our designated timeframe, number of articles in which the applicant was first author, the number of citations each paper received, the journal name, the impact factor of the journal (from the Scopus database), whether or not the papers were ophthalmology related, the type of paper (original research, case reports, or review articles), and the total h-index. The h-index is defined as the number of papers, h, that have ≥h citations each, and it has previously been used in the field of ophthalmology.[5]
[6]
[7]
[8]
[9] Original research included clinical and laboratory investigations. Approval from the Rutgers University Institutional Review Board was obtained prospectively.
Primary Outcome
The primary outcome examined was the tier of ophthalmology program matriculated into by each applicant. The programs were ranked on two metrics. The first metric (metric A) was based on a study published by Thiessen et al which ranked ophthalmology departments based on the sum and mean h-index of each faculty member in the respective instiution.[9] To evaluate whether research parameters are still significant when an alternate tiering modality is used, U.S. News and World Report Ophthalmology Rankings (also published in Doximity) were used to evaluate programs based on overall reputation (metric B). Although other ranking modalities are available such as the list published by Ophthalmology Times, only the list published by U.S. News and World Report ranks all the residency programs with regard to reputation, rather than just the top 10. Reputation, according to the U.S. News and World Report, is calculated annually via surveys sent to board-certified physicians who each nominate five residency programs in their respective specialties. These rankings are up to date but due to the methodology of surveys, they are also prone to bias. We hope to account for this limitation by using a more objective ranking system, as noted earlier. Residency programs using both metrics are tiered according to the following schema: tier 1 (residencies ranked 1–20), tier 2 (21–40), tier 3 (41–60), tier 4 (61–80), and tier 5 (remaining residencies). This schema follows the method published by Kashkoush et al.[4] Medical schools of applicants were also tiered in a similar format according to research output as reported by Thiessen et al and reputation published by the U.S. News and World Report Ophthalmology Rankings.[10]
Statistical Analysis
Statistical analysis was replicated using the method previously outlined by Kashkoush et al.[4] First, all bibliometric variables were tested for normal distribution using the Shapiro–Wilk test of normalcy. Next, univariate ordinal regression analysis was run. Ordinal regression was used to accommodate for the ranking method of tiers. For variables that were not normally distributed, Kruskal–Wallis test was used. Variables found to be significant on univariate analysis (p < 0.05) were evaluated for collinearity using variance inflation factors (VIF). As published in previous literature, when two variables have high VIF, signifying high collinearity, the variable that interferes the least with R-squared in linear regression can be removed.[11] After running VIF analysis, variables that were not collinear were input into a multivariate ordinal regression analysis. For the significant variables in the multivariate ordinal regression analysis, a post hoc Bonferroni test was performed to note for any differences between the different tiers. All statistical analyses were performed using Statistical Package for the Social Sciences (SPSS) version 22 (IBM, Armonk, NY).
Results
The final cohort included 340 accepted applicants from a total of 464 accepted applicants (73%) from 87 out of 117 (74%) ophthalmology residency programs.[12] Baseline characteristics of research parameters are listed in [Table 1]. A total of 340 applicants had a total of 421 publications, for an average of 1.23 ± 0.01 standard error of mean (SEM) papers per applicant and the number of publications ranged from 0 to 44 publications.
Table 1
Overview of resident publication characteristics
|
Variable
|
Value (mean ± standard error of mean)
|
|
Publications/Resident
|
1.23 ± 0.01
|
|
Highest journal impact factor
|
4.20 ± 0.58
|
|
Average journal impact factor
|
2.76 ± 0.41
|
|
Hirsch's index
|
2.44 ± 0.69
|
|
Years since first publication
|
2.70 ± 2.71
|
|
Total number (percentage)
|
|
MD/PhD
|
15 (4.2)
|
|
Ophthalmology-related publications
|
228 (54)
|
|
First-author publications
|
125 (30)
|
|
Original research articles
|
323 (77)
|
|
Review papers
|
37 (9)
|
|
Case reports
|
61 (14)
|
|
Five most common journals for publications
|
Total publications per journal (% of total publications)
|
|
IOVS
|
40 (18)
|
|
Retina
|
18 (8)
|
|
Molecular Vision
|
12 (5)
|
|
JAMA
|
11 (5)
|
|
OSLI Retina
|
11 (5)
|
Abbreviations: IOVS, Investigative Ophthalmology and Visual Science; JAMA, The Journal of the American Medical Association; OSLI, Ophthalmic Surgery, Lasers and Imaging.
The publication count and total h-index were right skewed in distribution. Of the 340 applicants, 88 matched into tier 1 residency programs (26%), 69 in tier 2 (20%), 38 in tier 3 (11%), 57 in tier 4 (17%), and 88 in tier 5 (26%).
Of the variables measured, there were four variables associated with matching at a higher tier program based on research output (metric A) on univariate analysis ([Table 2]): total number of articles published (p = 0.01), highest journal impact factor (p < 0.0001), average journal impact factor (p < 0.0001), and total h-index (p = 0.04). Five variables on univariate analysis were associated with matching at a higher tier program based on reputation (metric B; [Table 2]). VIFs were performed to eliminate variables with high multicollinearity, defined in literature as VIF greater than 5 ([Tables 3] and [4]).[11] On multivariate ordinal regression, only h-index was significantly associated with tier of matriculation based on metric A (p < 0.0001); however, no variable was associated with tier of matriculation based on metric B.
Table 2
Univariate and multivariate analysis of publication characteristics for residents stratified by residency program tier based on institutional research output (A) and based on institutional reputation (B)
|
Variable
|
Tier of residency based on institutional research output[a]
|
Value (mean ± SEM)
|
p-Value (univariate)
|
p-Value (multivariate)
|
Tier of residency based on reputation[a]
|
Value (mean ± SEM)
|
p-Value (univariate)
|
p-Value (multivariate)
|
|
Total number of articles published
|
1
|
1.57 ± 0.34
|
0.010[b]
|
0.502
|
1
|
2.11 ± 0.56
|
<0.0001[b]
|
0.168
|
|
2
|
1.63 ± 0.71
|
|
|
2
|
0.79 ± 0.21
|
|
|
|
3
|
0.53 ± 0.26
|
|
|
3
|
0.51 ± 0.14
|
|
|
|
4
|
0.48 ± 0.12
|
|
|
4
|
0.80 ± 0.23
|
|
|
|
5
|
0.52 ± 0.16
|
|
|
5
|
1.11 ± 0.51
|
|
|
|
Number of first-author papers
|
1
|
0.51 ± 0.12
|
0.330
|
Not included
|
1
|
0.60 ± 0.15
|
0.053
|
Not included
|
|
2
|
0.46 ± 0.16
|
|
|
2
|
0.39 ± 0.12
|
|
|
|
3
|
0.32 ± 0.16
|
|
|
3
|
0.14 ± 0.07
|
|
|
|
4
|
0.09 ± 0.05
|
|
|
4
|
0.24 ± 0.10
|
|
|
|
5
|
0.18 ± 0.08
|
|
|
5
|
0.34 ± 0.20
|
|
|
|
Original research
|
1
|
1.13 ± 0.31
|
0.386
|
Not included
|
1
|
1.81 ± 0.50
|
0.018
|
Not included
|
|
2
|
1.27 ± 0.58
|
|
|
2
|
0.65 ± 0.19
|
|
|
|
3
|
0.62 ± 0.27
|
|
|
3
|
0.36 ± 0.11
|
|
|
|
4
|
0.41 ± 0.11
|
|
|
4
|
0.67 ± 0.20
|
|
|
|
5
|
0.41 ± 0.14
|
|
|
5
|
0.74 ± 0.34
|
|
|
|
Number of case reports
|
1
|
0.15 ± 0.05
|
0.878
|
Not included
|
1
|
0.23 ± 0.09
|
0.080
|
Not included
|
|
2
|
0.25 ± 0.14
|
|
|
2
|
0.07 ± 0.05
|
|
|
|
3
|
0.06 ± 0.06
|
|
|
3
|
0.14 ± 0.07
|
|
|
|
4
|
0.04 ± 0.02
|
|
|
4
|
0.06 ± 0.06
|
|
|
|
5
|
0.09 ± 0.04
|
|
|
5
|
0.21 ± 0.09
|
|
|
|
Number of review papers
|
1
|
0.15 ± 0.06
|
0.326
|
Not included
|
1
|
0.19 ± 0.07
|
0.108
|
Not included
|
|
2
|
0.16 ± 0.09
|
|
|
2
|
0.09 ± 0.05
|
|
|
|
3
|
0.03 ± 0.03
|
|
|
3
|
–
|
|
|
|
4
|
0.04 ± 0.02
|
|
|
4
|
0.07 ± 0.06
|
|
|
|
5
|
0.02 ± 0.02
|
|
|
5
|
0.08 ± 0.04
|
|
|
|
Number of ophthalmology-related papers
|
1
|
0.70 ± 0.17
|
0.919
|
Not included
|
1
|
1.02 ± 0.25
|
0.194
|
Not included
|
|
2
|
0.72 ± 0.29
|
|
|
2
|
0.46 ± 0.18
|
|
|
|
3
|
0.65 ± 0.30
|
|
|
3
|
0.30 ± 0.11
|
|
|
|
4
|
0.34 ± 0.10
|
|
|
4
|
0.41 ± 0.16
|
|
|
|
5
|
0.29 ± 0.12
|
|
|
5
|
0.90 ± 0.43
|
|
|
|
Highest journal impact factor
|
1
|
2.44 ± 0.57
|
<0.0001[b]
|
Not included
|
1
|
2.52 ± 0.55
|
0.023[b]
|
0.392
|
|
2
|
2.00 ± 0.65
|
|
|
2
|
1.76 ± 0.74
|
|
|
|
3
|
0.92 ± 0.32
|
|
|
3
|
0.91 ± 0.25
|
|
|
|
4
|
1.26 ± 0.33
|
|
|
4
|
1.07 ± 0.28
|
|
|
|
5
|
0.67 ± 0.21
|
|
|
5
|
1.03 ± 0.29
|
|
|
|
Average journal impact factor
|
1
|
1.46 ± 0.23
|
<0.0001[b]
|
0.902
|
1
|
1.40 ± 0.23
|
0.070
|
Not included
|
|
2
|
1.12 ± 0.27
|
|
|
2
|
0.97 ± 0.27
|
|
|
|
3
|
0.76 ± 0.28
|
|
|
3
|
0.79 ± 0.22
|
|
|
|
4
|
1.17 ± 0.31
|
|
|
4
|
0.88 ± 0.22
|
|
|
|
5
|
0.49 ± 0.14
|
|
|
5
|
0.78 ± 0.22
|
|
|
|
Hirsch's index
|
1
|
1.67 ± 0.27
|
0.037[b]
|
0.002[b]
|
1
|
1.32 ± 0.27
|
0.010[b]
|
0.874
|
|
2
|
1.19 ± 0.26
|
|
|
2
|
0.75 ± 0.20
|
|
|
|
3
|
0.79 ± 0.28
|
|
|
3
|
0.33 ± 0.11
|
|
|
|
4
|
0.54 ± 0.15
|
|
|
4
|
0.69 ± 0.21
|
|
|
|
5
|
0.51 ± 0.14
|
|
|
5
|
0.62 ± 0.23
|
|
|
|
Years since first publication
|
1
|
1.22 ± 0.25
|
0.185
|
Not included
|
1
|
3.03 ± 0.41
|
0.024[b]
|
0.715
|
|
2
|
0.70 ± 0.23
|
|
|
2
|
2.07 ± 0.45
|
|
|
|
3
|
0.56 ± 0.29
|
|
|
3
|
1.62 ± 0.32
|
|
|
|
4
|
0.57 ± 0.17
|
|
|
4
|
2.06 ± 0.40
|
|
|
|
5
|
0.52 ± 0.16
|
|
|
5
|
1.58 ± 0.37
|
|
|
Abbreviation: SEM, standard error of mean.
a Tier of program based on U.S. News and World Report Ophthalmology Rankings.
b Statistically significant values.
Notes: Univariate analysis was used to determine if each variable significantly affected the residency tier. Multivariate analysis was then performed using all significant variables. p-Values were calculated by univariate and multivariate analyses and were significant if p < 0.05.
Table 3
Dependent variable: tier of matriculation based on research output
|
Variable
|
VIF before
|
VIF after
|
|
Average journal impact factor
|
3.727
|
1.627
|
|
Hirsch's index total
|
2.907
|
2.773
|
|
Highest journal impact factor
|
5.140
|
–
|
|
Total number of publications
|
2.073
|
2.055
|
Abbreviation: VIF, variance inflation factor.
Notes: VIF measured for variables significant on univariate analysis. “VIF before” indicates before multicollinearity diagnostics were taken into account. “VIF after” indicates calculation of multicollinearity after removing variables that were highly collinear.
Table 4
Dependent variable: tier of matriculation based on reputation
|
Variable
|
VIF before
|
VIF after
|
|
Total number of publications
|
6.028
|
2.214
|
|
Total number of original research articles
|
7.509
|
–
|
|
Highest journal impact factor
|
2.172
|
2.171
|
|
Hirsch's index
|
5.108
|
4.406
|
|
Years since first publication
|
2.948
|
2.936
|
Abbreviation: VIF, variance inflation factor.
Notes: VIF measured for variables significant on univariate analysis. “VIF before” indicates before multicollinearity diagnostics were taken into account. “VIF after” indicates calculation of multicollinearity after removing variables that were highly collinear.
Since h-index was the only variable significant on multivariate analysis, linear regression analysis was used to identify which research parameters significantly impacted h-index. Inputs tested include the independent variables listed in [Table 5]. The variables overall accounted for 78% of the h-index variability (p < 0.0001). Of these variables, the ones that were significant included the number of articles for which the applicant was first author (p < 0.0001), number of ophthalmology-related papers (p < 0.0001), average journal impact (p < 0.0001), and years since first publication (p < 0.0001). The h-index was noted to increase by 1 for every 3.1 papers as the first author, every 4.9 years since the first publication, every 6.4 ophthalmology-related publications, and every publication in a journal with an impact factor of 5.2. The number of case reports was negatively correlated with the h-index (p < 0.0001).
Table 5
Measuring the effect of publication variables on an applicant's Hirsch's index using linear regression
|
|
|
Confidence interval (95%) of beta coefficient
|
|
Variable
|
Beta[a]
|
p-Value
|
Lower bound
|
Upper bound
|
|
Total number published
|
0.477
|
0.064
|
−0.016
|
0.582
|
|
Number of first-author publications
|
0.318
|
<0.0001
|
0.402
|
0.848
|
|
Number of original articles
|
−0.108
|
−0.612
|
−0.367
|
0.216
|
|
Number of case reports
|
−0.284
|
<0.0001
|
−1.200
|
−0.577
|
|
Number of review papers
|
−0.026
|
0.623
|
−0.531
|
0.319
|
|
Number of ophthalmology-related papers
|
0.155
|
<0.0001
|
0.074
|
0.296
|
|
Average impact factor
|
0.198
|
<0.0001
|
0.090
|
0.293
|
|
Highest impact factor
|
−0.023
|
0.684
|
−0.065
|
0.043
|
|
Years since first publication
|
0.203
|
<0.001
|
0.117
|
0.306
|
|
Tier of medical school based on research output
|
0.009
|
0.886
|
0.070
|
1.113
|
|
Tier of medical school based on reputation
|
0.234
|
0.189
|
0.170
|
2.103
|
a Beta coefficients quantify the sensitivity of the change that the independent variable (listed under “Variable”) has on the dependent variable (Hirsch's index).
Given the geographical impact of the ophthalmology match,[1] we were interested to see if the h-index was different in applicants who matched at their home program. Of the 340 applicants, 63 matched into their home program and 277 matched at a different program. Applicants who matched in programs not affiliated with their medical school had a significantly higher h-index (1.16 ± 0.26 SEM) than those who matched at their home program (0.63 ± 0.03 SEM; p = 0.02).
Furthermore, 15 of the 340 residents were listed as MD/PhDs. They had significantly higher h-indices (5.40 ± 0.78 SEM vs. 0.86 ± 0.08 SEM; p < 0.0001) and also matched in higher tier programs based on research output (3.00 ± 0.57 SEM vs. 2.13 ± 0.30 SEM; p = 0.02) when compared with applicants who had MDs. However, there was no significant difference in the tier of program they matched at when evaluating reputation of residency program (2.59 ± 0.42 SEM vs. 2.35 ± 0.23 SEM; p = 0.56).
Discussion
Although there have been multiple studies published regarding the important aspects of matching into ophthalmology, none of them have objectively analyzed the relationship between research and matching into an ophthalmology program.[2]
[13] Our results show that in terms of research, the most accurate predictor of getting into a top-tier research ophthalmology program is the h-index. Similar findings were observed in neurosurgery applicants.[4] This was true on multivariate analysis, after taking into account the total number of articles published and average journal impact factor.
On the contrary, no research parameter was significantly associated with matching into a top-tier program based on reputation. This highlights two important conclusions. First, the reputation of a residency program is not determined solely by scholarly productivity. Although overlap between the two ranking systems exists (the same 15 programs are seen in the top 20 of both lists, but in a different order), reputation also takes into account factors other than research. The metric used in the ranking system of Thiessen et al is based on the h-indices of an institution's faculty. The reputation, as determined by surveys sent out by U.S. News and World Report, is a compilation of an institution's scholarly productivity as well as caliber of residency training, mentorship, and the clinical expertise attained by its former attendees as evaluated by fellow ophthalmologists. Second, this finding also highlights that other factors might play an even more important role in successfully matriculating into a residency program ranked higher on the reputation scale. In the analysis of Driver et al, both letters of recommendation from ophthalmologists and U.S. Medical Licensing Examination (USMLE) had higher odds ratio than any research parameters evaluated including journal impact factor and total number of publications.[13] However, this association is to be taken with caution, since many applicants receive strong letter of recommendations from ophthalmologists by collaborating with them on research projects.
Only 54% of the papers published by the applicants were ophthalmology related and the number of ophthalmology-related publications did not correlate with tier of program matriculation, which is interesting given that research is sometimes used as a marker of an applicant's interest in the field.[14] Residency programs perhaps use research, irrespective of field, as an overall marker of an applicant's future interest in academics and research to both enhance the residency program and the future of the field. On average, applicants got their first papers published approximately 2.7 years before Electronic Residency Application Service (ERAS) was submitted. Therefore, it is possible that most of the applicants started research when they were not yet involved in ophthalmology research. Furthermore, the years since first publication significantly contributed to the h-index, which is expected as it allows more time for the publications to be cited. This association also suggests that participating in research early on, irrespective of field, can be beneficial to ophthalmology applicants.
Furthermore, the number of ophthalmology-related papers correlated with a higher h-index. It is possibly because the ophthalmology journals that applicants in our analysis were most likely to publish in ([Table 1]) were high-impact journals. Alternatively, the number of case reports was found to be inversely related to the h-index, possibly because case reports are not as commonly cited as original research papers and review articles. Nonresearch parameters such as an applicant's tier of medical school did not correlate with h-index. Top-tier institutions receive more research funding; however, this did not prevent applicants at lower tier medical schools from being productive in terms of research. The interdependence between external factors such as USMLE scores and h-index cannot be determined as this information is not publicly accessible. To our knowledge, there have been no studies to evaluate the academic profile of applicants with high research output. A study by Stain et al noted that research, defined in their study as number of publications, was independently associated with matching into a high-tier institution after accounting for USMLE scores and AOA membership; however, the relationship between these variables has yet to be studied.[15]
A significantly higher h-index was needed to match at a program that was not affiliated with the applicants' medical school. This correlates with previous studies noting that applicants with higher USMLE Step 1 scores had significantly higher odds of matching into an ophthalmology program in a different region as their medical school.[13] These findings highlight that a stronger applicant resume, both in relation to scores and research, provides an opportunity to attend residency programs away from their home institutions.
Applicants with a doctorate degree had significantly higher h-indices and were significantly more likely to match at higher tiered institutions based on institutional research output compared with applicants without a doctoral degree. This can be attributed to a research-oriented institution's interest in attracting MD/PhD candidates as well as the preference for MD/PhD candidates to want to attend an institution with increased access to research facilities and grants. However, having a doctoral degree did not relate to tier of matriculation based on reputation. According to a survey completed by program directors, chairpersons, and people involved in the match, 86% of them believed that a PhD was not important, or of little or average importance to match into ophthalmology.[3]
A limitation to this study was that we did not have access to complete data from the San Francisco Match, the regulating body for the Ophthalmology match. However, we were still able to obtain information on 73% of the residents in the Class of 2019. We were limited to the papers within the Scopus database and did not account for other modalities of research such as abstracts and presentations which applicants also have on their residency applications. Also, not all program Web sites listed if residents had dual degrees, which may have skewed the data. We were not able to assess the influence of other factors including AOA status, USMLE scores, letters of recommendation, etc., on the selection process or assess the relationship between these variables and research. Previous studies have examined the importance of these factors; however, the purpose of our study was to provide an in-depth analysis on what attributes of research are important to match into ophthalmology residency and how research influences tier of matriculation.[1] Finally, no one ranking system can accurately capture the tier of ophthalmology residency programs. We tried to overcome this limitation by using two ranking programs that utilized two different schemata to construct their lists.
In conclusion, these findings have important implications for applicants interested in ophthalmology. We found that applicants with more impactful research were more likely to match at higher tiered research institutions. However, research is not directly correlated with matching at a more reputable institution. Indirect associations may exist; for example, applicants with more extensive research may have more opportunities to interact with mentors who can provide stronger letters of recommendation, but these correlations are beyond the scope of our analysis. Overall, starting research early, publishing ophthalmology-related articles, being the first author, and publishing in established journals increase an applicant's h-index. This analysis can help advisors and applicants have a better understanding of the role research has in their application process.
