Subject and Academic Setting of Pre-residency Publications as Potential Predictors of Post-residency Academic Productivity in a Cohort of Ophthalmology Residents

Abstract

Objective This study aimed to determine if there is any association between the subject of pre-residency research publications (PRPs) and the academic settings in which they are produced with post-residency academic productivity among graduates from an academic ophthalmology residency program.

Design, Setting, Participants This is a cross-sectional study involving graduates of the Wilmer Eye Institute Residency Training Program from 1990 to 1999. An electronic survey was conducted and each participant was asked to submit his/her curriculum vitae, including a list of peer-reviewed publications. Publications were validated and then classified according to the academic setting in which the research was performed, and whether or not the research was related to ophthalmology.

Outcome Measures The primary outcome measure was the post-residency academic productivity score. The secondary outcome measures were the relationships of academic productivity with the settings in which PRPs were performed and the topics of the PRPs.

Results Fifty-one individuals were included. Regression analysis showed a positive association between the number of PRPs generated during undergraduate studies and medical school and the academic productivity score (ratio = 1.17, p = 0.006) but not during an advanced degree program, research fellowship, or a year off to perform research. Regardless of the setting in which it was performed, the subject of a PRP was not associated with academic productivity.

Conclusion It may be appropriate for ophthalmology residency programs whose mission is to train future academicians to place increased weight on applicants who have published articles related to projects performed during undergraduate years or in medical school and less weight on publications related to research performed during a more structured research period, such as an advanced degree program, regardless of the subject(s) of the publication(s).

Keywords

academic productivity - ophthalmology - residency - publications

The importance of an academic career in medicine cannot be overemphasized; physicians in the academic field become leaders in their profession. They are involved in research, education, and administrative duties, and they participate in various endeavors that promote the advancement of the field. The residency training program at the Wilmer Eye Institute of the Johns Hopkins Hospital, like other academic training programs, seeks applicants who are most likely to pursue an academic career after completing residency.

Predictors for a career choice in academics and academic productivity have been studied in various medical specialties[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] but, until recently, not in ophthalmology.[13] [14] [15] Pre-residency characteristics associated with a post-residency career choice in academic ophthalmology include completion of a PhD degree, a long record of published peer-reviewed scientific papers, and an interest in neurology as an alternate career choice.[15] A previous study by our group identified several pre-residency characteristics that positively correlated with post-residency academic productivity, including membership in the Alpha Omega Alpha (AOA) honor society and the total number of first- and second-authored research publications in peer-reviewed scientific journals, whereas older age at the start of residency was negatively correlated with post-residency academic productivity.[14] Other factors such as medical school class rank and medical licensure exam scores were not found to be predictive.[14]

The number of pre-residency research publications (PRPs) has been associated with post-residency academic productivity not only in ophthalmology but also in neurology,[1] neurosurgery,[2] plastic surgery,[3] and radiology.[4] However, it is unclear if the settings in which the PRPs are written or the subject of the PRPs are specifically predictive of future academic productivity. For example, are publications produced during undergraduate study and/or medical school—settings in which they are not part of the curriculum—more predictive of future academic productivity than publications produced as a requirement for a PhD or Master's degree, formal research fellowship, or a year taken off to perform research? Also, are pre-residency published works in ophthalmology more predictive than research performed in other fields of science or medicine? Using the PubMed database, we were unable to find any studies that addressed these issues. Thus, our study was performed to determine if there were a correlation between the settings in which the research was performed, or the subject of the PRP (i.e., ophthalmology vs. nonophthalmology), and post-residency academic productivity among ophthalmology residency graduates from an academic institution. We hypothesized that individuals with PRPs prepared during undergraduate study or medical school—where research publications are not a requirement—would be more academically productive after residency than individuals who had fewer PRPs during the same academic periods. We also hypothesized that individuals with research publications during completion of a pre-residency Master's or PhD program, a formal research fellowship, or a year dedicated to research prior to starting residency—settings in which research publications are generally expected—would be no more likely to be academically productive than individuals with fewer PRPs. Lastly, we hypothesized that individuals with PRPs on ophthalmologic topics would be more academically productive than individuals who had fewer PRPs in the field of ophthalmology.

Methods

This study was approved by the Johns Hopkins Medical Institutional Review Board. It involved the same cohort of 57 individuals who graduated from the Wilmer Eye Institute Ophthalmology Residency Program between 1990 and 1999 that was surveyed and reported by Shan et al.[14] The complete methodology was detailed in that publication, and only the summary is presented here. Fifty-three graduates (93%) had contact information and were emailed a 16-question electronic survey. The survey included questions on pre-residency personal and scholastic information such as membership in the AOA medical honor society, United States Medical Licensing Examination (USMLE) 1 or Federal Licensing Examination (FLEX) percentile score, medical school rank, prior completion of a PhD, Master's program, research fellowship, or a year off performing research. Other items in the survey include questions regarding post-residency academic productivity, including faculty status, number of active grants and grants in submission. The graduates were asked to return the survey along with their most updated curriculum vitae (CV) that included a complete record of their formal education, post-residency employment, and a list of all peer-reviewed publications.

Only articles published before the start of residency were included. PRPs listed in CVs were cross-validated against the PubMed database to make certain that the applicants had not misrepresented their publication records as has been noted in other subspecialties.[16] [17] PRPs not found in the PubMed database were identified directly in the respective journals in which they were published (n = 4). Book chapter contributions and published abstracts were not included. Abstracts or full articles of all PRPs were retrieved and reviewed. The order of authorship for each research publication was noted; that is, first, second, middle, and last. Each publication was classified according to the academic setting in which it was undertaken: undergraduate, medical school, other advanced degree program (i.e., PhD or Master's), or year-off for those who took time off to do research or worked as a research fellow prior to residency. Publications were also classified as to whether or not the subject was related to the field of ophthalmology.

Recognizing that first authorship in research publications usually signifies substantial contribution to the research, we also performed a subanalysis of only first-authored PRPs, classifying them according to the settings in which they were produced and the subjects of each publication.

We used the same primary outcome measure of academic productivity used by Shan et al in their previous study[14]: an academic productivity score that was computed for each graduate using a point system that took into account activities important to successful physician-scientists, including faculty status, applying for and holding grants, mentoring pre-doctoral students and/or residents, contributing to the academic community as an article reviewer and/or editorial board member for one or more scientific journals, and being a national or international lecturer. For faculty status, graduates were categorized into one of three groups: currently on full-time faculty, previously on full-time faculty, or never on full-time faculty.

We looked for any association between the academic settings in which the PRPs were generated and whether or not the research was in the field of ophthalmology with the academic productivity score and faculty status.

Statistical Analysis

Frequency tables by faculty status were generated for the following PRP variables: number of research publications during post-graduate degree programs (PhD, Master's) and year-off, number of publications during undergraduate study and medical school, pre-residency first-authored publications, and PRPs in the field of ophthalmology versus other fields. For each pre-residency characteristic, multiple regression models appropriate for the data distributions were used to determine independent associations between the characteristic and outcome measure. Robust standard error estimates were used in the regression model. Specifically, multinomial logistic regression was used to examine the association between pre-residency characteristics and current faculty status, and multiple log-linear regression analysis was used for the productivity score. Due to the size of the sample, the regression model explored the association between each PRP predictor and the outcome, adjusting only for the graduates' age at the start of residency and AOA status, factors that were statistically significant in a previous paper involving this cohort of individuals.[14] All statistical analyses were performed using STATA software version 11 (StatCorp, College Station, TX). Nominally, significance was defined as p < 0.05.

Results

Fifty-three (53) individuals were contacted and 52 (98%) completed the survey. A list of PRPs was available for 51 individuals. The details of pre-residency characteristics and post-residency academic activities of this cohort were described in a previous publication.[14]

Prior to residency, 30 individuals (58%) had publications that were ophthalmology-related, whereas 28 (54%) had publications in a field of science or medicine other than ophthalmology. Thirty-three individuals (65%) had publications during undergraduate study, medical school, or both. Nine (18%) either had taken time off to do research or had a formal research fellowship prior to residency; all of these individuals had publications related to projects performed during the inclusive period. Seventeen (33%) individuals had other advanced degrees prior to residency; 14 (83%) of them had published research projects during their programs. Two individuals who had other advanced degrees but did not publish any research work during their programs finished a Master of Arts in Advanced International Studies and a Master's degree in medical sciences, respectively. A third individual did a simultaneous 3-year research fellowship in glaucoma and completed a Master's degree in public health; all of his publications (n = 18) were related to projects performed during the research fellowship. The same individual did not have any publications during undergraduate and medical schools. He was considered an outlier and was excluded from the final regression analysis.

Twenty-four individuals (47%) had at least one first-authored PRP in the field of ophthalmology, whereas 14 individuals (28%) had at least one first-authored, non-ophthalmology-related PRP.

The distribution of the subjects of the PRPs and the settings in which they were performed according to faculty status are shown in [Table 1]. The associations between pre-residency characteristics and current faculty status are presented in [Table 2]. None of the predictors for faculty status reached statistical significance; however, there was a trend toward positive association between individuals with more first-authored ophthalmology-related PRPs and current full-time faculty status (relative risk ratio = 2.54, confidence interval = 1.0–6.4, p = 0.05).

Table 1 Subject and setting of pre-residency publications by current faculty status (N = 51)

Characteristics	Never on full-time faculty (N = 13)	Once was on full-time faculty (N = 15)	Currently on full-time faculty (N = 23)	Total (N = 51)	p-Value[a]
Ophthalmology publications					0.31
Median (IQR)	0 (2)	1 (2)	2 (3)	1 (3)
Range	0–6	0–4	0–18	0–18
Nonophthalmology publications					0.55
Median (IQR)	1 (2)	0 (1)	1 (3)	1 (2)
Range	0–6	0–6	0–7	0–7
Publications during PhD/Master's program/year-off					0.67
Median (IQR)	0 (2)	0 (1)	0 (4)	0 (3)
Range	0–6	0–4	0–18	0–18
Publications during MD/pre-med					0.93
Median (IQR)	1 (1)	1(2)	1 (3)	1 (2)
Range	0–5	0–3	0–7	0–7
First-authored publications related to ophthalmology					0.55
Median (IQR)	0 (1)	0 (1)	1 (2)	0 (2)
Range	0–6	0–2	0–9	0–9
First-authored publications not related to ophthalmology					0.37
Median (IQR)	0 (2)	0 (1)	0 (0)	0(1)
Range	0–4	0–3	0–3	0–3

Abbreviation: IQR, interquartile range.

^a p-Values are from Kruskal–Wallis tests.

Table 2 Adjusted relative risk ratios of pre-residency publications for faculty status[a]

Faculty category	Characteristics	Relative risk ratio	95% Confidence interval	p-Value
Currently on full-time faculty vs. never on full-time faculty	Ophthalmology publications	1.51	0.9, 2.5	0.10
	Nonophthalmology publications	1.21	0.8, 1.8	0.36
	Publications during PhD/Master's program/year-off	1.35	0.9, 2.0	0.14
	Publications during MD/pre-med	1.24	0.8, 2.0	0.40
	First-authored publications related to ophthalmology	2.54	1.0, 6.4	0.05
	First-authored publications not related to ophthalmology	0.61	0.3, 1.3	0.20
Previously was on full-time faculty vs. never on full-time faculty	Ophthalmology publications	1.08	0.6, 2.0	0.79
	Nonophthalmology publications	1.10	0.7, 1.8	0.69
	Publications during PhD/Master's program/year-off	0.85	0.5, 1.6	0.60
	Publications during MD/pre-med	0.83	0.5, 1.3	0.40
	First-authored publications related to ophthalmology	1.66	0.6, 4.6	0.33
	First-authored publications not related to ophthalmology	0.76	0.3, 1.7	0.49

^a The adjusted estimates are from multivariate multinomial logistic regressions adjusting for age at the start of residency and Alpha Omega Alpha (AOA) status.

The associations between pre-residency characteristics and academic productivity score are shown in [Table 3]. There is a statistically significant association between the total number of PRPs generated from projects performed during undergraduate study and medical school and the academic productivity score (ratio = 1.17, p = 0.006). Individuals with PRPs during these periods on average had 1.17 times higher academic productivity scores than individuals who had no PRPs during the same academic periods. There was no such association for individuals who published peer-reviewed articles as part of a pre-residency advanced degree course, a formal research fellowship, or during a year off to pursue research. Although it did not reach statistical significance, individuals with PRPs in ophthalmology had higher academic productivity scores compared with individuals without PRPs in ophthalmology (ratio = 1.13, p = 0.06). The other variables were not found to be associated with academic productivity score.

Table 3 Adjusted ratios of pre-residency publications for academic productivity score from log-linear regression models[b]

Characteristics	Ratio[a]	95% Confidence interval	p
Ophthalmology publications	1.13	1.0, 1.3	0.06
Nonophthalmology publications	1.12	0.9, 1.3	0.20
Publications during PhD/Master's program/year-off	1.00	0.9, 1.1	0.97
Publications during MD/pre-med	1.17	1.0, 1.3	0.006
First-authored publications related to ophthalmology	1.04	0.9, 1.2	0.69
First-authored publications not related to ophthalmology	1.08	0.8, 1.6	0.65

^a Ratio represents the exponentiated coefficient from the log-linear regression model.

^b The models adjusted for age at the start of residency and Alpha Omega Alpha (AOA).

Discussion

Several studies in various subspecialties in medicine have attempted to identify potential predictors of a productive academic career. Some have assessed pre-residency characteristics, whereas others have investigated the residency training factors that may predict a successful academic career. Although we recognize that there are many factors during residency training that can influence an individual's career choice, we decided to look into purely pre-residency characteristics. Results of this study may provide guidance to admission committees of academic ophthalmology programs in the selection of individuals who not only have the potential but also are most likely to pursue an academic career after residency. This study aimed to assess whether or not characteristics of pre-residency, peer-reviewed scientific publications are predictive of subsequent academic productivity in ophthalmology.

The total number of PRPs has been shown to be a positive predictor for an academic career in many studies that involved graduates from both single[1] [3] [4] [15] and multiple training institutions.[2] Shan et al reported that, in the same cohort of individuals as the present study, there was a trend toward a positive correlation between the number of first- and second-authored PRPs and the number of post-residency research publications (IRR = 1.14, p = 0.08).[14] The current study included all PRPs regardless of authorship and also classified each PRP according to the academic setting in which the research was performed, and whether or not the PRP was related to the field of ophthalmology. In addition, we performed a subgroup analysis of first-authored PRPs, as first-authored research publications have been shown to be associated with successful future academic careers in neurosurgery.[12] We found that only PRPs related to projects performed during undergraduate years or medical school were associated with academic productivity as measured by our scoring system, whereas neither the subject of the PRP nor the order of authorship was associated with future academic productivity in our cohort of ophthalmologists.

As there is no single, universally accepted measure for academic productivity,[11] [13] we opted to use two different outcome measures for academic productivity: faculty status and academic productivity score. The academic productivity score takes into account various achievements and activities that are important for a successful academic career, such as post-residency research publications, faculty status, grants, editorial board membership, reviewer appointment, and mentorship. Therefore, we believe it is a reasonable outcome measure of academic productivity.

Our results show that the number of peer-reviewed scientific publications related to research undertaken during undergraduate study and medical school is positively associated with the academic productivity score. Individuals who were authors of PRPs in these settings had, on average, 17% higher academic productivity scores than individuals who did not generate PRPs during the same academic periods. This finding supports the concept of encouraging participation in research activities among college and medical students. It also suggests that individuals who perform research in ophthalmology during medical school are truly interested in an academic career, not just in being identified as a “better” candidate for residency than a peer who performed no such research. Indeed, we believe that early exposure to research not only cultivates a student's interest in scientific pursuits but also provides the student with many of the skills required of a clinician-scientist. This is consistent with the findings of several studies that show that research experience gained prior to residency contributes to an individual's decision and capacity to perform subsequent research.[7] [18] In a systematic review of scholarly articles across all specialties of medicine published from 1990 to 2005, participation in research projects that culminated to publications in peer-reviewed scientific journals during medical school was determined to be associated with selection of an academic career.[18] Grim and colleagues reported that undergraduate school and medical school publications were independently predictive of future academic career among radiology residency applicants.[19] In a survey among 318 medical school graduates, respondents who performed extracurricular research published more after graduation compared with students without such experience. Furthermore, medical students who published an article as a result of the extracurricular research experience had more research publications after graduation compared with students who had research experience but no publications.[20] It is plausible that earlier research involvement may stem from early interest in an academic career and that this experience further inspires these individuals to be academically productive after completing their training.

Although we found a definite correlation between PRPs published during college or medical school and academic productivity, we found no such correlation between PRPs published as part of advanced degree programs, formal research fellowships, or during a year off to pursue research.

Field of Medicine in Which the Research-Related PRPs Were Performed and Post-residency Academic Productivity

Although the number of PRPs on ophthalmologic topics showed a trend toward a positive association with the academic productivity score, this did not reach statistical significance, and there was no association between the number of PRPs on nonophthalmic topics and future academic productivity. Caution should be applied in interpreting these results, as PRPs in ophthalmology may suggest an individual's interest in an academic career in ophthalmology.

This study has several limitations including those previously enumerated by Shan and colleagues.[14] The methodology employed in this study is susceptible to high recall bias. The list of PRPs was obtained from the respective CV submitted by each individual. Nevertheless, each PRP was validated against the PubMed database. In addition, this study included a cohort of individuals who graduated from a single institution. This population may not be an accurate representation of all ophthalmology residency graduates in the United States. Lastly, H-index, a popular measure of academic productivity, was not used in the study. We instead adopted the scoring system that was used in the first paper from our group.[14] This scoring system aimed to be more comprehensive, as it took into account other scholarly activities of a clinician-scientist in addition to the publication record and faculty status.

Conclusion

The results of this study show that, at least with respect to the residents in our program, the total number of PRPs produced during undergraduate studies and medical school is positively correlated with subsequent academic productivity, whereas there is no such association between the total number of PRPs produced during a PhD or Master's program, during a formal research fellowship, or during a year away from school to pursue research. Although the factors that influence the decision of a particular individual as to whether or not to pursue an academic career are multiple and complex, our findings suggest that it may be appropriate for ophthalmology residency programs whose mission is to train future academicians to place increased weight on applicants who have published articles related to projects performed during undergraduate years or in medical school and less weight on publications related to research performed during a more structured research period, such as an advanced degree program, regardless of the subject(s) of the publication(s).

Conflict of Interest

None declared.

Authors' Contributions

Dr. Cruz was involved in acquisition, analysis, and interpretation of data, and drafting and revising the manuscript. Ms. Wang was involved in analysis and interpretation of the data and revising of the manuscript. Dr. Joseph was involved in the design of the work, acquisition, and interpretation of the data and revising the manuscript. Dr. Miller was involved in design of the research, interpretation of the data, revising the manuscript, and final approval of the version to be published.

Ethical Approval

This study was approved by the Johns Hopkins Medical Institutional Review Board.

• References

• 1 Dorsey ER, Raphael BA, Balcer LJ, Galetta SL. Predictors of future publication record and academic rank in a cohort of neurology residents. Neurology 2006; 67 (08) 1335-1337
• 2 McClelland III S. Pre-residency peer-reviewed publications are associated with neurosurgery resident choice of academic compared to private practice careers. J Clin Neurosci 2010; 17 (03) 287-289
• 3 Grewal NS, Spoon DB, Kawamoto HK. , et al. Predictive factors in identifying subspecialty fellowship applicants who will have academic practices. Plast Reconstr Surg 2008; 122 (04) 1264-1271 , discussion 1272–1273
• 4 Rezek I, McDonald RJ, Kallmes DF. Pre-residency publication rate strongly predicts future academic radiology potential. Acad Radiol 2012; 19 (05) 632-634
• 5 Brancati FL, Mead LA, Levine DM, Martin D, Margolis S, Klag MJ. Early predictors of career achievement in academic medicine. JAMA 1992; 267 (10) 1372-1376
• 6 Ferrer RL, Katerndahl DA. Predictors of short-term and long-term scholarly activity by academic faculty: a departmental case study. Fam Med 2002; 34 (06) 455-461
• 7 Ledley FD, Lovejoy Jr FH. Factors influencing the interests, career paths, and research activities of recent graduates from an academic, pediatric residency program. Pediatrics 1993; 92 (03) 436-441
• 8 Patterson SK, Fitzgerald JT, Boyse TD, Cohan RH. Is past academic productivity predictive of radiology resident academic productivity?. Acad Radiol 2002; 9 (02) 211-216
• 9 Daly KA, Levine SC, Adams GL. Predictors for resident success in otolaryngology. J Am Coll Surg 2006; 202 (04) 649-654
• 10 Markert RJ, Part HM, Vetter DK. Predictors of careers in academic medicine for graduates of a community-based, primary-care-oriented medical school. Acad Med 1998; 73 (07) 790-793
• 11 Leng T, Jaben KA, Zheng L. , et al. The influence of medical school and residency training program upon choice of an academic career and academic productivity among otolaryngology faculty in the United States of America: our experience of 1433 academic otolaryngologists. Clin Otolaryngol 2012; 37 (01) 58-62
• 12 Crowley RW, Asthagiri AR, Starke RM, Zusman EE, Chiocca EA, Lonser RR. ; Research Committee of the Congress of Neurological Surgeons. In-training factors predictive of choosing and sustaining a productive academic career path in neurological surgery. Neurosurgery 2012; 70 (04) 1024-1032
• 13 Lee AG, Golnik KC, Oetting TA. , et al. Re-engineering the resident applicant selection process in ophthalmology: a literature review and recommendations for improvement. Surv Ophthalmol 2008; 53 (02) 164-176
• 14 Shan SJC, Wang J, West-Gower E, Miller NR. Pre-residency characteristics associated with post-residency academic productivity in a cohort of ophthalmology residents. J Acad Ophthalmol 2012; 5 (01) 24-36
• 15 Pineles SL, Galetta SL, Fine SL. , et al. Predictive characteristics and factors influencing career choice amongst ophthalmology trainees. J Acad Ophthalmol 2011; 4: 73-84
• 16 Baker DR, Jackson VP. Misrepresentation of publications by radiology residency applicants. Acad Radiol 2000; 7 (09) 727-729
• 17 Dale JA, Schmitt CM, Crosby LA. Misrepresentation of research criteria by orthopaedic residency applicants. J Bone Joint Surg Am 1999; 81 (12) 1679-1681
• 18 Straus SE, Straus C, Tzanetos K. ; International Campaign to Revitalise Academic Medicine. Career choice in academic medicine: systematic review. J Gen Intern Med 2006; 21 (12) 1222-1229
• 19 Grim LJ, Shapiro LM, Singhapricha T, Mazurowski MA, Desser TS, Maxfield CM. Predictors of an academic career on radiology residency applications. Acad Radiol 2014; 21 (05) 685-690
• 20 Reinders JJ, Kropmans TJB, Cohen-Schotanus J. Extracurricular research experience of medical students and their scientific output after graduation. Med Educ 2005; 39 (02) 237