Keywords
ophthalmology residency - medical school - research - bibliometrics - h-index - residency
match
Many factors influence an applicant's matriculation into an ophthalmology residency
program. Previous studies have assessed the importance of quantifiable aspects of
the application, such as board scores, Alpha Omega Alpha (AOA) status, and clinical
grades in matching into ophthalmology.[1]
[2] Although research was mentioned in several of these studies, its effects on the
match process has never been quantified. A survey of program directors, chairpersons,
and people involved in matching committees indicated that a year of research was noted
to be helpful in improving ophthalmology applications.[3]
In this study, we analyze the correlation between different research parameters and
the rank of residency program matched, as measured by research output and reputation.
Similar bibliometric studies have been done to evaluate the impact of research on
matching into neurosurgery.[4] An objective analysis of the often-elusive term “research” can help medical students
and advisors understand the impact of research on their residency application. We
aim to create an accurate research profile of ophthalmology applicants by analyzing
the papers published by current ophthalmology residents while they were applying to
residency. An inherent limitation to our approach of primarily relying on the number
of total publications is that it fails to fully capture the research contribution
of medical students as listed on their San Francisco Match residency applications,
which also include abstracts, presentations, and manuscripts in submission. However,
access to this information is not readily available, and by focusing on the quality
and quantity of published manuscripts we hope to provide a more uniform, objective
analysis as to how research affects the chances of matching into ophthalmology.
Methods
Setting and Study Population
First, 117 residency programs using the public Accreditation Council for Graduate
Medical Education (ACGME) database were identified (https://apps.acgme.org/ads/public/reports/report/1 with “Ophthalmology” as the specialty. Accessed September 24, 2017). Next, we obtained
the names of the members of the Ophthalmology Class of 2019 from individual program
Web sites. If this information was not verified, it was omitted from the analysis.
Thirty programs which did not list their residents were omitted from the data analysis.
From the Web sites, we noted each resident's name, medical school, and degree type.
After compiling a list of the applicants, we used the “Author Search” option on Scopus
to look up each applicant individually. When an applicant's name did not yield any
results, it was counted as a 0 for published papers and Hirsch's index (h-index) in
the analysis. If results were unclear for an applicant, they were omitted from analysis.
In total, eight applicants were removed for missing information, resulting in a final
cohort of 340 applicants. For each profile, the number of published papers accepted
on or prior to September 2014 were noted, to best reflect the number of fully published
manuscripts each applicant reported to the San Francisco match.
For each applicant search on Scopus, we noted the number of papers published within
our designated timeframe, number of articles in which the applicant was first author,
the number of citations each paper received, the journal name, the impact factor of
the journal (from the Scopus database), whether or not the papers were ophthalmology
related, the type of paper (original research, case reports, or review articles),
and the total h-index. The h-index is defined as the number of papers, h, that have
≥h citations each, and it has previously been used in the field of ophthalmology.[5]
[6]
[7]
[8]
[9] Original research included clinical and laboratory investigations. Approval from
the Rutgers University Institutional Review Board was obtained prospectively.
Primary Outcome
The primary outcome examined was the tier of ophthalmology program matriculated into
by each applicant. The programs were ranked on two metrics. The first metric (metric
A) was based on a study published by Thiessen et al which ranked ophthalmology departments
based on the sum and mean h-index of each faculty member in the respective instiution.[9] To evaluate whether research parameters are still significant when an alternate
tiering modality is used, U.S. News and World Report Ophthalmology Rankings (also
published in Doximity) were used to evaluate programs based on overall reputation
(metric B). Although other ranking modalities are available such as the list published
by Ophthalmology Times, only the list published by U.S. News and World Report ranks all the residency programs
with regard to reputation, rather than just the top 10. Reputation, according to the
U.S. News and World Report, is calculated annually via surveys sent to board-certified
physicians who each nominate five residency programs in their respective specialties.
These rankings are up to date but due to the methodology of surveys, they are also
prone to bias. We hope to account for this limitation by using a more objective ranking
system, as noted earlier. Residency programs using both metrics are tiered according
to the following schema: tier 1 (residencies ranked 1–20), tier 2 (21–40), tier 3
(41–60), tier 4 (61–80), and tier 5 (remaining residencies). This schema follows the
method published by Kashkoush et al.[4] Medical schools of applicants were also tiered in a similar format according to
research output as reported by Thiessen et al and reputation published by the U.S.
News and World Report Ophthalmology Rankings.[10]
Statistical Analysis
Statistical analysis was replicated using the method previously outlined by Kashkoush
et al.[4] First, all bibliometric variables were tested for normal distribution using the
Shapiro–Wilk test of normalcy. Next, univariate ordinal regression analysis was run.
Ordinal regression was used to accommodate for the ranking method of tiers. For variables
that were not normally distributed, Kruskal–Wallis test was used. Variables found
to be significant on univariate analysis (p < 0.05) were evaluated for collinearity using variance inflation factors (VIF). As
published in previous literature, when two variables have high VIF, signifying high
collinearity, the variable that interferes the least with R-squared in linear regression
can be removed.[11] After running VIF analysis, variables that were not collinear were input into a
multivariate ordinal regression analysis. For the significant variables in the multivariate
ordinal regression analysis, a post hoc Bonferroni test was performed to note for
any differences between the different tiers. All statistical analyses were performed
using Statistical Package for the Social Sciences (SPSS) version 22 (IBM, Armonk,
NY).
Results
The final cohort included 340 accepted applicants from a total of 464 accepted applicants
(73%) from 87 out of 117 (74%) ophthalmology residency programs.[12] Baseline characteristics of research parameters are listed in [Table 1]. A total of 340 applicants had a total of 421 publications, for an average of 1.23 ± 0.01
standard error of mean (SEM) papers per applicant and the number of publications ranged
from 0 to 44 publications.
Table 1
Overview of resident publication characteristics
|
Variable
|
Value (mean ± standard error of mean)
|
|
Publications/Resident
|
1.23 ± 0.01
|
|
Highest journal impact factor
|
4.20 ± 0.58
|
|
Average journal impact factor
|
2.76 ± 0.41
|
|
Hirsch's index
|
2.44 ± 0.69
|
|
Years since first publication
|
2.70 ± 2.71
|
|
Total number (percentage)
|
|
MD/PhD
|
15 (4.2)
|
|
Ophthalmology-related publications
|
228 (54)
|
|
First-author publications
|
125 (30)
|
|
Original research articles
|
323 (77)
|
|
Review papers
|
37 (9)
|
|
Case reports
|
61 (14)
|
|
Five most common journals for publications
|
Total publications per journal (% of total publications)
|
|
IOVS
|
40 (18)
|
|
Retina
|
18 (8)
|
|
Molecular Vision
|
12 (5)
|
|
JAMA
|
11 (5)
|
|
OSLI Retina
|
11 (5)
|
Abbreviations: IOVS, Investigative Ophthalmology and Visual Science; JAMA, The Journal
of the American Medical Association; OSLI, Ophthalmic Surgery, Lasers and Imaging.
The publication count and total h-index were right skewed in distribution. Of the
340 applicants, 88 matched into tier 1 residency programs (26%), 69 in tier 2 (20%),
38 in tier 3 (11%), 57 in tier 4 (17%), and 88 in tier 5 (26%).
Of the variables measured, there were four variables associated with matching at a
higher tier program based on research output (metric A) on univariate analysis ([Table 2]): total number of articles published (p = 0.01), highest journal impact factor (p < 0.0001), average journal impact factor (p < 0.0001), and total h-index (p = 0.04). Five variables on univariate analysis were associated with matching at a
higher tier program based on reputation (metric B; [Table 2]). VIFs were performed to eliminate variables with high multicollinearity, defined
in literature as VIF greater than 5 ([Tables 3] and [4]).[11] On multivariate ordinal regression, only h-index was significantly associated with
tier of matriculation based on metric A (p < 0.0001); however, no variable was associated with tier of matriculation based on
metric B.
Table 2
Univariate and multivariate analysis of publication characteristics for residents
stratified by residency program tier based on institutional research output (A) and
based on institutional reputation (B)
|
Variable
|
Tier of residency based on institutional research output[a]
|
Value (mean ± SEM)
|
p-Value (univariate)
|
p-Value (multivariate)
|
Tier of residency based on reputation[a]
|
Value (mean ± SEM)
|
p-Value (univariate)
|
p-Value (multivariate)
|
|
Total number of articles published
|
1
|
1.57 ± 0.34
|
0.010[b]
|
0.502
|
1
|
2.11 ± 0.56
|
<0.0001[b]
|
0.168
|
|
2
|
1.63 ± 0.71
|
|
|
2
|
0.79 ± 0.21
|
|
|
|
3
|
0.53 ± 0.26
|
|
|
3
|
0.51 ± 0.14
|
|
|
|
4
|
0.48 ± 0.12
|
|
|
4
|
0.80 ± 0.23
|
|
|
|
5
|
0.52 ± 0.16
|
|
|
5
|
1.11 ± 0.51
|
|
|
|
Number of first-author papers
|
1
|
0.51 ± 0.12
|
0.330
|
Not included
|
1
|
0.60 ± 0.15
|
0.053
|
Not included
|
|
2
|
0.46 ± 0.16
|
|
|
2
|
0.39 ± 0.12
|
|
|
|
3
|
0.32 ± 0.16
|
|
|
3
|
0.14 ± 0.07
|
|
|
|
4
|
0.09 ± 0.05
|
|
|
4
|
0.24 ± 0.10
|
|
|
|
5
|
0.18 ± 0.08
|
|
|
5
|
0.34 ± 0.20
|
|
|
|
Original research
|
1
|
1.13 ± 0.31
|
0.386
|
Not included
|
1
|
1.81 ± 0.50
|
0.018
|
Not included
|
|
2
|
1.27 ± 0.58
|
|
|
2
|
0.65 ± 0.19
|
|
|
|
3
|
0.62 ± 0.27
|
|
|
3
|
0.36 ± 0.11
|
|
|
|
4
|
0.41 ± 0.11
|
|
|
4
|
0.67 ± 0.20
|
|
|
|
5
|
0.41 ± 0.14
|
|
|
5
|
0.74 ± 0.34
|
|
|
|
Number of case reports
|
1
|
0.15 ± 0.05
|
0.878
|
Not included
|
1
|
0.23 ± 0.09
|
0.080
|
Not included
|
|
2
|
0.25 ± 0.14
|
|
|
2
|
0.07 ± 0.05
|
|
|
|
3
|
0.06 ± 0.06
|
|
|
3
|
0.14 ± 0.07
|
|
|
|
4
|
0.04 ± 0.02
|
|
|
4
|
0.06 ± 0.06
|
|
|
|
5
|
0.09 ± 0.04
|
|
|
5
|
0.21 ± 0.09
|
|
|
|
Number of review papers
|
1
|
0.15 ± 0.06
|
0.326
|
Not included
|
1
|
0.19 ± 0.07
|
0.108
|
Not included
|
|
2
|
0.16 ± 0.09
|
|
|
2
|
0.09 ± 0.05
|
|
|
|
3
|
0.03 ± 0.03
|
|
|
3
|
–
|
|
|
|
4
|
0.04 ± 0.02
|
|
|
4
|
0.07 ± 0.06
|
|
|
|
5
|
0.02 ± 0.02
|
|
|
5
|
0.08 ± 0.04
|
|
|
|
Number of ophthalmology-related papers
|
1
|
0.70 ± 0.17
|
0.919
|
Not included
|
1
|
1.02 ± 0.25
|
0.194
|
Not included
|
|
2
|
0.72 ± 0.29
|
|
|
2
|
0.46 ± 0.18
|
|
|
|
3
|
0.65 ± 0.30
|
|
|
3
|
0.30 ± 0.11
|
|
|
|
4
|
0.34 ± 0.10
|
|
|
4
|
0.41 ± 0.16
|
|
|
|
5
|
0.29 ± 0.12
|
|
|
5
|
0.90 ± 0.43
|
|
|
|
Highest journal impact factor
|
1
|
2.44 ± 0.57
|
<0.0001[b]
|
Not included
|
1
|
2.52 ± 0.55
|
0.023[b]
|
0.392
|
|
2
|
2.00 ± 0.65
|
|
|
2
|
1.76 ± 0.74
|
|
|
|
3
|
0.92 ± 0.32
|
|
|
3
|
0.91 ± 0.25
|
|
|
|
4
|
1.26 ± 0.33
|
|
|
4
|
1.07 ± 0.28
|
|
|
|
5
|
0.67 ± 0.21
|
|
|
5
|
1.03 ± 0.29
|
|
|
|
Average journal impact factor
|
1
|
1.46 ± 0.23
|
<0.0001[b]
|
0.902
|
1
|
1.40 ± 0.23
|
0.070
|
Not included
|
|
2
|
1.12 ± 0.27
|
|
|
2
|
0.97 ± 0.27
|
|
|
|
3
|
0.76 ± 0.28
|
|
|
3
|
0.79 ± 0.22
|
|
|
|
4
|
1.17 ± 0.31
|
|
|
4
|
0.88 ± 0.22
|
|
|
|
5
|
0.49 ± 0.14
|
|
|
5
|
0.78 ± 0.22
|
|
|
|
Hirsch's index
|
1
|
1.67 ± 0.27
|
0.037[b]
|
0.002[b]
|
1
|
1.32 ± 0.27
|
0.010[b]
|
0.874
|
|
2
|
1.19 ± 0.26
|
|
|
2
|
0.75 ± 0.20
|
|
|
|
3
|
0.79 ± 0.28
|
|
|
3
|
0.33 ± 0.11
|
|
|
|
4
|
0.54 ± 0.15
|
|
|
4
|
0.69 ± 0.21
|
|
|
|
5
|
0.51 ± 0.14
|
|
|
5
|
0.62 ± 0.23
|
|
|
|
Years since first publication
|
1
|
1.22 ± 0.25
|
0.185
|
Not included
|
1
|
3.03 ± 0.41
|
0.024[b]
|
0.715
|
|
2
|
0.70 ± 0.23
|
|
|
2
|
2.07 ± 0.45
|
|
|
|
3
|
0.56 ± 0.29
|
|
|
3
|
1.62 ± 0.32
|
|
|
|
4
|
0.57 ± 0.17
|
|
|
4
|
2.06 ± 0.40
|
|
|
|
5
|
0.52 ± 0.16
|
|
|
5
|
1.58 ± 0.37
|
|
|
Abbreviation: SEM, standard error of mean.
a Tier of program based on U.S. News and World Report Ophthalmology Rankings.
b Statistically significant values.
Notes: Univariate analysis was used to determine if each variable significantly affected
the residency tier. Multivariate analysis was then performed using all significant
variables. p-Values were calculated by univariate and multivariate analyses and were significant
if p < 0.05.
Table 3
Dependent variable: tier of matriculation based on research output
|
Variable
|
VIF before
|
VIF after
|
|
Average journal impact factor
|
3.727
|
1.627
|
|
Hirsch's index total
|
2.907
|
2.773
|
|
Highest journal impact factor
|
5.140
|
–
|
|
Total number of publications
|
2.073
|
2.055
|
Abbreviation: VIF, variance inflation factor.
Notes: VIF measured for variables significant on univariate analysis. “VIF before”
indicates before multicollinearity diagnostics were taken into account. “VIF after”
indicates calculation of multicollinearity after removing variables that were highly
collinear.
Table 4
Dependent variable: tier of matriculation based on reputation
|
Variable
|
VIF before
|
VIF after
|
|
Total number of publications
|
6.028
|
2.214
|
|
Total number of original research articles
|
7.509
|
–
|
|
Highest journal impact factor
|
2.172
|
2.171
|
|
Hirsch's index
|
5.108
|
4.406
|
|
Years since first publication
|
2.948
|
2.936
|
Abbreviation: VIF, variance inflation factor.
Notes: VIF measured for variables significant on univariate analysis. “VIF before”
indicates before multicollinearity diagnostics were taken into account. “VIF after”
indicates calculation of multicollinearity after removing variables that were highly
collinear.
Since h-index was the only variable significant on multivariate analysis, linear regression
analysis was used to identify which research parameters significantly impacted h-index.
Inputs tested include the independent variables listed in [Table 5]. The variables overall accounted for 78% of the h-index variability (p < 0.0001). Of these variables, the ones that were significant included the number
of articles for which the applicant was first author (p < 0.0001), number of ophthalmology-related papers (p < 0.0001), average journal impact (p < 0.0001), and years since first publication (p < 0.0001). The h-index was noted to increase by 1 for every 3.1 papers as the first
author, every 4.9 years since the first publication, every 6.4 ophthalmology-related
publications, and every publication in a journal with an impact factor of 5.2. The
number of case reports was negatively correlated with the h-index (p < 0.0001).
Table 5
Measuring the effect of publication variables on an applicant's Hirsch's index using
linear regression
|
|
|
Confidence interval (95%) of beta coefficient
|
|
Variable
|
Beta[a]
|
p-Value
|
Lower bound
|
Upper bound
|
|
Total number published
|
0.477
|
0.064
|
−0.016
|
0.582
|
|
Number of first-author publications
|
0.318
|
<0.0001
|
0.402
|
0.848
|
|
Number of original articles
|
−0.108
|
−0.612
|
−0.367
|
0.216
|
|
Number of case reports
|
−0.284
|
<0.0001
|
−1.200
|
−0.577
|
|
Number of review papers
|
−0.026
|
0.623
|
−0.531
|
0.319
|
|
Number of ophthalmology-related papers
|
0.155
|
<0.0001
|
0.074
|
0.296
|
|
Average impact factor
|
0.198
|
<0.0001
|
0.090
|
0.293
|
|
Highest impact factor
|
−0.023
|
0.684
|
−0.065
|
0.043
|
|
Years since first publication
|
0.203
|
<0.001
|
0.117
|
0.306
|
|
Tier of medical school based on research output
|
0.009
|
0.886
|
0.070
|
1.113
|
|
Tier of medical school based on reputation
|
0.234
|
0.189
|
0.170
|
2.103
|
a Beta coefficients quantify the sensitivity of the change that the independent variable
(listed under “Variable”) has on the dependent variable (Hirsch's index).
Given the geographical impact of the ophthalmology match,[1] we were interested to see if the h-index was different in applicants who matched
at their home program. Of the 340 applicants, 63 matched into their home program and
277 matched at a different program. Applicants who matched in programs not affiliated
with their medical school had a significantly higher h-index (1.16 ± 0.26 SEM) than
those who matched at their home program (0.63 ± 0.03 SEM; p = 0.02).
Furthermore, 15 of the 340 residents were listed as MD/PhDs. They had significantly
higher h-indices (5.40 ± 0.78 SEM vs. 0.86 ± 0.08 SEM; p < 0.0001) and also matched in higher tier programs based on research output (3.00 ± 0.57
SEM vs. 2.13 ± 0.30 SEM; p = 0.02) when compared with applicants who had MDs. However, there was no significant
difference in the tier of program they matched at when evaluating reputation of residency
program (2.59 ± 0.42 SEM vs. 2.35 ± 0.23 SEM; p = 0.56).
Discussion
Although there have been multiple studies published regarding the important aspects
of matching into ophthalmology, none of them have objectively analyzed the relationship
between research and matching into an ophthalmology program.[2]
[13] Our results show that in terms of research, the most accurate predictor of getting
into a top-tier research ophthalmology program is the h-index. Similar findings were
observed in neurosurgery applicants.[4] This was true on multivariate analysis, after taking into account the total number
of articles published and average journal impact factor.
On the contrary, no research parameter was significantly associated with matching
into a top-tier program based on reputation. This highlights two important conclusions.
First, the reputation of a residency program is not determined solely by scholarly
productivity. Although overlap between the two ranking systems exists (the same 15
programs are seen in the top 20 of both lists, but in a different order), reputation
also takes into account factors other than research. The metric used in the ranking
system of Thiessen et al is based on the h-indices of an institution's faculty. The
reputation, as determined by surveys sent out by U.S. News and World Report, is a
compilation of an institution's scholarly productivity as well as caliber of residency
training, mentorship, and the clinical expertise attained by its former attendees
as evaluated by fellow ophthalmologists. Second, this finding also highlights that
other factors might play an even more important role in successfully matriculating
into a residency program ranked higher on the reputation scale. In the analysis of
Driver et al, both letters of recommendation from ophthalmologists and U.S. Medical
Licensing Examination (USMLE) had higher odds ratio than any research parameters evaluated
including journal impact factor and total number of publications.[13] However, this association is to be taken with caution, since many applicants receive
strong letter of recommendations from ophthalmologists by collaborating with them
on research projects.
Only 54% of the papers published by the applicants were ophthalmology related and
the number of ophthalmology-related publications did not correlate with tier of program
matriculation, which is interesting given that research is sometimes used as a marker
of an applicant's interest in the field.[14] Residency programs perhaps use research, irrespective of field, as an overall marker
of an applicant's future interest in academics and research to both enhance the residency
program and the future of the field. On average, applicants got their first papers
published approximately 2.7 years before Electronic Residency Application Service
(ERAS) was submitted. Therefore, it is possible that most of the applicants started
research when they were not yet involved in ophthalmology research. Furthermore, the
years since first publication significantly contributed to the h-index, which is expected
as it allows more time for the publications to be cited. This association also suggests
that participating in research early on, irrespective of field, can be beneficial
to ophthalmology applicants.
Furthermore, the number of ophthalmology-related papers correlated with a higher h-index.
It is possibly because the ophthalmology journals that applicants in our analysis
were most likely to publish in ([Table 1]) were high-impact journals. Alternatively, the number of case reports was found
to be inversely related to the h-index, possibly because case reports are not as commonly
cited as original research papers and review articles. Nonresearch parameters such
as an applicant's tier of medical school did not correlate with h-index. Top-tier
institutions receive more research funding; however, this did not prevent applicants
at lower tier medical schools from being productive in terms of research. The interdependence
between external factors such as USMLE scores and h-index cannot be determined as
this information is not publicly accessible. To our knowledge, there have been no
studies to evaluate the academic profile of applicants with high research output.
A study by Stain et al noted that research, defined in their study as number of publications,
was independently associated with matching into a high-tier institution after accounting
for USMLE scores and AOA membership; however, the relationship between these variables
has yet to be studied.[15]
A significantly higher h-index was needed to match at a program that was not affiliated
with the applicants' medical school. This correlates with previous studies noting
that applicants with higher USMLE Step 1 scores had significantly higher odds of matching
into an ophthalmology program in a different region as their medical school.[13] These findings highlight that a stronger applicant resume, both in relation to scores
and research, provides an opportunity to attend residency programs away from their
home institutions.
Applicants with a doctorate degree had significantly higher h-indices and were significantly
more likely to match at higher tiered institutions based on institutional research
output compared with applicants without a doctoral degree. This can be attributed
to a research-oriented institution's interest in attracting MD/PhD candidates as well
as the preference for MD/PhD candidates to want to attend an institution with increased
access to research facilities and grants. However, having a doctoral degree did not
relate to tier of matriculation based on reputation. According to a survey completed
by program directors, chairpersons, and people involved in the match, 86% of them
believed that a PhD was not important, or of little or average importance to match
into ophthalmology.[3]
A limitation to this study was that we did not have access to complete data from the
San Francisco Match, the regulating body for the Ophthalmology match. However, we
were still able to obtain information on 73% of the residents in the Class of 2019.
We were limited to the papers within the Scopus database and did not account for other
modalities of research such as abstracts and presentations which applicants also have
on their residency applications. Also, not all program Web sites listed if residents
had dual degrees, which may have skewed the data. We were not able to assess the influence
of other factors including AOA status, USMLE scores, letters of recommendation, etc.,
on the selection process or assess the relationship between these variables and research.
Previous studies have examined the importance of these factors; however, the purpose
of our study was to provide an in-depth analysis on what attributes of research are
important to match into ophthalmology residency and how research influences tier of
matriculation.[1] Finally, no one ranking system can accurately capture the tier of ophthalmology
residency programs. We tried to overcome this limitation by using two ranking programs
that utilized two different schemata to construct their lists.
In conclusion, these findings have important implications for applicants interested
in ophthalmology. We found that applicants with more impactful research were more
likely to match at higher tiered research institutions. However, research is not directly
correlated with matching at a more reputable institution. Indirect associations may
exist; for example, applicants with more extensive research may have more opportunities
to interact with mentors who can provide stronger letters of recommendation, but these
correlations are beyond the scope of our analysis. Overall, starting research early,
publishing ophthalmology-related articles, being the first author, and publishing
in established journals increase an applicant's h-index. This analysis can help advisors
and applicants have a better understanding of the role research has in their application
process.
