Recruiting the Digital-Age Applicant: The Impact of Ophthalmology Residency Program Web Presence on Residency Recruitment

Abstract

Background The phenomenon of internet dependence has changed the way the rising generation seeks information. This mentality has caused medical students to turn to online resources as they seek information about potential residency training programs. Residency program web presence (PWP) is increasingly important, and may even impact recruitment efforts. Improvement of PWP could enhance programs' recruitment of ideal candidates.

Objectives The purpose of this study is to assess how ophthalmology residency PWP is impacting the residency recruitment process by understanding how it influences applicants' application and rank list choices as well as to identify the contributing factors.

Methods Applicants applying for ophthalmology residency training at Penn State University during the 2015–2016 and 2016–2017 application cycles were surveyed using Research Electronic Data Capture (REDCap). Surveys sought applicants' perspectives with respect to their experiences with PWP and how those experiences shaped their application and rank list decisions.

Results Of 860 applicants, 214 (24.9%) responded, accounting for 17.4% (214/1,228) of all ophthalmology residency applicants during the respective cycles; 72.4% of respondents expressed PWP does impact where they apply, how they form their rank list, or both; 93.4% said websites are an important resource during the application process; 47.2% conveyed interest in programs utilizing social media tools; and 76.5% of respondents felt websites gave sufficient information less than 50% of the time.

Conclusion Ophthalmology PWP does impact resident recruitment. By enhancing program websites and adding social media tools, programs can improve recruitment efforts.

Keywords

ophthalmology residency - website - social media - web presence - recruitment - residency application - residency match

This is the age of incessant digital information. People of all age groups are continually increasing their use of internet and social media networks as technology reliance explodes.[1] [2] For upper-level medical students, this means increased dependence on residency program web presence (PWP), which includes program websites, social media applications, and other online tools, when gathering information about residency programs during the application process.[3] [4] [5] In addition, recent studies suggest the use of social media tools, such as Twitter and Facebook, permeate medical student preferences for residency program information dissemination.[6] [7]

Though program websites may not be the most important factor in recruitment, they do impact applicants' decisions.[8] Studies in various specialties have found that the information available on residency program websites varies greatly.[9] [10] Due to varying amounts of information and inconsistent user-friendliness, students have been disillusioned by the quality of online resources.[6] Other studies have concluded that programs are underutilizing these tools to reach out to and inform prospective applicants.[7] [11] [12]

In ophthalmology, investigations have separately evaluated the existence and content of residency program websites, and factors important in recruitment of applicants.[10] [13] However, to our knowledge, no study has evaluated the role of PWP in resident recruitment. The purpose of this study is to begin to understand whether PWP is impacting the residency recruitment process by influencing where applicants apply and how they form rank lists and, if so, to identify contributing factors.

Methods

Study Design

Using Research Electronic Data Capture (REDCap), a mixed model, cross-sectional, anonymous response survey was sent, via email link, to medical students applying for ophthalmology residency near the conclusion of their residency interview/match process. All surveyed applicants applied to the ophthalmology residency program at the Pennsylvania State University for consideration in the 2015–2016 or 2016–2017 cycles. Each group was queried during their respective application period, near rank list submission. Survey questions explored the impact PWP had on their application and match process, what factors had the most impact, obstacles they encountered, and where they sought information about programs. Skip logic was used to ensure that respondents only answered those items that pertained to related previous responses.

Five questions required participants to “rank” items. This survey applied the rank system convention that a lower value is superior to a higher value. Three questions required ranking of program factors that may impact recruitment. The factors considered were additional application requirements, benefits, call schedules, conference schedules, contact information of program, community outreach opportunities, curriculum, deadlines for application, FAQ (frequently asked questions), facility information, faculty information, fellowship match history, international opportunities, interview dates, location, outside activities (leisure), perceived reputation, program philosophy, recommendation by faculty or resident, resident reviews/comments, research opportunities, resources, rotation schedules, surgical numbers, statistics (average OKAP scores, etc.), VA information, and other. Applicants were asked to rank their “top 5” of these factors, as they pertained to each question.

Two questions had respondents consider the usefulness of various social media platforms (Facebook, Twitter, Doximity, Student Doctor Network [SDN], and other) and preference of additional resources used (word of mouth, faculty, online databases, program coordinator or direct, and residents). Participants ranked all items in these questions from most to least.

Approval of this study was obtained from the Pennsylvania State University Internal Review Board.

Data Analysis

Generally, responses were analyzed based on summary statistics and response percentages. However, when a ranking of options was requested, the total number of ranks, the mean rank, and a weighted rank were calculated. The weighted rank was used to lend credence to both the mean rank and total number of ranks when comparing each answer choice. Weighted ranks were calculated as follows:

Weight rank = mean rank × [1–(total rank/total items ranked)].

A weighted rank with a lower value is calculated to be superior to a higher value. In the rank comparisons, only options that received at least 10% of the sample's vote were considered significant. This eliminated the risk that the formula poses of having an item that was rarely ranked from outweighing an item that received a significant number of responses.

An inductive thematic analysis was performed on applicants' comments with regard to poorly navigable and user-friendly websites by two of the authors (M.G-J., M.M.).[14] Based on the results of this review, themes with assigned numeric codes were established and the authors reviewed responses individually to assign a theme to each response. The authors then discussed the coding to come to an agreement on thematic assignments.

Results

The response rate was 214/860 (24.9%). This accounts for 17.4% of the 1,228 total applicants who submitted rank lists for ophthalmology residency during the 2015–2016 and 2016–2017 application cycles.[15]

Impact of Web Presence

With regard to the influence of PWP, 106 (49.53%) respondents noted it impacts only where they apply, 13 (6.07%) responded it impacts only their rank list, and 36 (16.8%) indicated it impacts both where they apply and how they rank a program. Fifty-nine (27.6%) indicated that web presence does not influence where they apply or how they make their rank list.

Of respondents, 201 (93.4%) expressed websites are an important resource during the application process. One hundred and three (48.1%) noted social media as a helpful tool for programs and 101 (47.2%) would like to see an increase in the use of social media tools for dissemination of program information. The order of preferred social media tools (from most to least useful for applicants) was SDN, Doximity, Facebook, Twitter, and “other” ([Table 1]).

Table 1 Applicants' web tool preferences, in addition to websites

	Weighted rank	Mean rank	Total ranks
Most useful social media platforms for applicants
Student Doctor Network (SDN)	0.60407	2.63679	212
Doximity	0.63642	2.73460	211
Facebook	0.70048	2.91866	209
Twitter	0.79120	3.29665	209
Other[a]	2.44868	2.97959	49
Other resources utilized by applicants to learn about residency programs
Word of mouth	0.83795	2.95431	197
Faculty at your institution	0.85449	2.97449	196
Residents at your institution	1.04556	3.26738	187
Online databases (e.g., FREIDA)	1.18703	3.62703	185
Program coordinator of the program of interest	1.67326	4.38235	170
Program director of the program of interest	1.72062	4.54971	171
Other[b]	3.42139	4.35593	59

^a Written responses receiving more than one comment: Instagram (7), LinkedIn (6), matchapplicants.com (4), program website (4), Google (2), YouTube (2), SF match (2), and additional information via email (2).

^b Written responses receiving more than one comment: SDN/online forums (20), faculty and residents at other institution (7), Doximity (2), program website (2), and Google (2).

Supplemental resources used by respondents, in addition to websites and social media, to learn about programs were ranked, in order of use, as follows: word of mouth, faculty at their home institution, residents at their home institution, online databases (e.g., FREIDA), program coordinator of the program of interest, program director of the program of interest, and “other” ([Table 1]).

Quality of Program Web Presence

When asked about user-friendliness and navigability of websites, 183 (85.5%) respondents indicated sites met those characteristics only “sometimes”; 19 (8.9%) said sites were “rarely” navigable and friendly; and 12 (5.6%) responded sites were “always” navigable and friendly. Three themes emerged when text responses for “rarely” were examined: exhaustive and misleading searches (9), lack of updated and relevant content (8), and broken links (2) ([Table 2]).

Table 2 Reasons applicants find websites to be rarely user friendly and navigable

Applicant's comment	Theme[a]
Information is extremely vague	3
Information not up to date, links often don't work, websites difficult to find	2
It is not easy to find the links I'm looking for such as links to the curriculum. Sometimes the links do not lead to pages I'm looking for. For example, “education” is not always graduate medical education, sometimes it means patient education	1
Lack of information or information that is cached in an odd or difficult-to-find location	2
Lengthy unnecessary information posted. We need concise information about surgical volume and fellowship placements	2
Most websites include generic infos with no specifics about what are you looking for in a candidate leaving vague impressions. Websites should have specific useful criteria	2
Often there are many hidden trees to get to relevant information. I find myself having to click on every single permutation of links to ensure that I find all the information I want	2
Old, outdated info	3
Outdated information, advertisement language rather than program facts/surgical numbers, etc.	3
Relevant information usually requires digging	2
Several steps to find “education” or “residency program.” Faculty profiles and program descriptions outdated and highlight “nonessential” on website	2
Sometimes the links do not work or the information seems outdated	1
They often reveal little about the faculty and current residents	3
They seem outdated	3
Because they do not have important universal info such as surgical numbers, fellowship matches, current resident info (med school)	3
Hard to find residency site versus regular ophthalmology department site	2
Hard to find things	2
Not a lot of information available	3
Outdated content, generic information, no real sense of the unique aspects of the program	3

^a Themes: 1—broken links, 2—exhaustive and misleading searches, 3—lack of updated relevant content.

Seventy-one (33.2%) respondents indicated websites gave adequate information more than 50% of the time, while 143 (76.8%) respondents noted information to be adequate 50% or less of the time. One hundred and ninety-nine (93.0%) felt there were insufficient resident reviews and comments in PWP.

Important Factors in Recruitment

The factors most likely to determine if respondents applied to a program were, in order, surgical numbers, location, perceived reputation, recommendation by faculty or resident, and fellowship match history ([Table 3]).

Table 3 Factors determining application submission

	Weighted rank	Mean rank	Total ranks[a]
Factors most likely to determine whether an applicant applies to a residency program
Surgical numbers	1.54805	2.85714	126
Location(s)	1.57726	2.76271	118
Perceived reputation	1.64423	2.64423	104
Other	1.76727	1.80000	5
Recommendation by faculty or resident	1.83089	2.67816	87
Fellowship match history	1.99969	3.02151	93
Additional requirements (stereo vision, statement of interest, etc.)	2.09455	2.18182	11
Program philosophy	2.14319	2.75410	61
Deadlines for application	2.16000	2.20000	5
Curriculum	2.18366	2.94366	71
Contact information of program	2.21727	2.25000	4
Conference schedules	2.55273	2.60000	5
Interview dates	2.77152	2.83333	6
Positions available per year	2.89943	3.28125	32
Resources available to residents	2.90667	3.0303	33
Research opportunities	2.90681	3.56863	51
Faculty information	2.93333	2.33333	33
VA information	2.96242	3.13333	15
Benefits	3.00476	3.15385	13
Call schedules	3.02295	3.37931	29
Rotation schedules	3.04242	3.33333	24
Community outreach opportunities	3.18623	3.35714	14
Facility information	3.23750	3.43750	16
Statistics (OKAP scores, etc.)	3.25403	3.42857	14
International opportunities	3.42273	3.75000	24
FAQ	3.64955	3.87500	16
Outside activities (leisure)	3.84000	4.00000	11

^a Only responses with greater than 10% (21) response rate considered significant.

Whether a respondent ranked a program was most heavily influenced, in order, by location, surgical numbers, perceived reputation, program philosophy, and resident reviews/comments ([Table 4]).

Table 4 Factors that determine whether or not a program is ranked

	Weighted rank	Mean rank	Total ranks[a]
Contact information of the program	0.99636	1.00000	1
Location	1.57141	2.73504	117
Surgical numbers	1.69908	2.95726	117
Perceived reputation	1.79955	2.79592	98
Program philosophy	1.80283	2.55556	81
Other	1.80818	1.95000	20
Resident reviews/comments	1.92642	2.78824	85
Interview dates	2.16000	2.20000	5
Curriculum	2.19000	2.75000	56
Fellowship match history	2.20388	3.25843	89
Recommendation by faculty or resident	2.38088	3.01724	58
Resources available to residents	2.63081	3.05263	38
Faculty information	2.63865	2.92593	27
Outside activities (leisure, etc.)	2.71161	2.84615	13
Research opportunities	2.88406	3.41860	43
Positions available per year	3.02545	3.20000	15
Rotation schedules	3.03529	3.23529	17
Statistics (OKAP scores, etc.)	3.08848	3.26667	15
Facility information	3.20208	3.28571	7
Benefits	3.64091	3.75000	8
International opportunities	3.64235	3.88235	17
VA information	3.69545	3.75000	4
Call schedules	3.70023	4.18750	32
Community outreach opportunities	3.94182	4.00000	4
Conference schedules	3.95636	4.00000	3
FAQ	3.97091	4.00000	2
Deadlines for application	–	–	Not ranked
Additional requirements (stereo vision, statement of interest, etc.)	–	–	Not ranked

^a Only responses with greater than 10% (21) response rate considered significant.

Respondents said the most difficult items to find online were resident reviews/comments, surgical numbers, call schedules, program philosophy, and fellowship match history ([Table 5]).

Table 5 Residency program information that is most difficult to find

	Weighted rank	Mean rank	Total ranks[a]
Resident reviews/comments	1.18997	2.51724	145
Surgical numbers	1.26956	2.45865	133
Other	1.97091	2.00000	4
Call schedules	1.97834	2.98925	93
Program philosophy	2.22258	2.57895	38
Fellowship match history	2.38990	3.30263	76
Positions available per year	2.46364	2.50000	4
Resources available to residents	2.48763	3.19672	61
Interview dates	2.52424	2.83333	30
Curriculum	2.54988	2.78261	23
FAQ	2.70773	2.87500	16
International opportunities	2.72153	3.15789	38
Outside activities (leisure, etc.)	2.72153	3.15789	38
Facility information	2.73818	3.00000	24
Community outreach opportunities	2.77727	3.25000	40
Conference schedules	2.80364	3.00000	18
Research opportunities	2.96000	3.36364	33
Rotation schedules	2.96417	3.38235	34
VA information	2.98814	3.26087	23
Faculty information	3.07286	3.40741	27
Benefits	3.12034	3.53125	32
Contact information of program	3.27636	3.40000	10
Additional requirements (stereo vision, statement of interest, etc.)	3.31123	3.52941	17
Locations	3.33818	3.40000	5
Deadlines for application	4.12364	4.20000	5
Statistics (average board score, etc.)	–	–	Not ranked

^a Only responses with greater than 10% (21) response rate considered significant.

Discussion

Our survey indicates that PWP does impact recruitment. A majority of respondents, 155 (72.4%), expressed PWP impacted where they applied, how they formed their rank list, or both, which suggests applicants are relying heavily on web tools throughout the interview and match processes. Such a reliance on web tools by applicants is not surprising, as dependence on the internet resources is continually increasing.[8] Similar trends with residency program websites have also been established in other specialty programs.[3] [4] [5] Furthermore, there is considerable interest among applicants for social media use by programs, though it did not reach a majority in our sample (47.2%). Interestingly, if programs decide to use social media, their presence in this arena appears to be more important than the particular tool employed, as evidenced by the relatively even distribution of responses between SDN, Doximity, Facebook, and Twitter. As ophthalmology programs move forward, they should recognize the impact their PWP has on recruitment efforts. Understanding applicants' preferences with respect to web tools will allow programs to better address the needs of prospective residents.

Unfortunately, ophthalmology applicants feel program websites are frequently difficult to use, with all respondents indicating websites are unfriendly and unnavigable to some degree. Notably, some information applicants indicate as being most impactful, such as resident reviews, program philosophy, fellowship match details, and surgical volume are also items that are among the most difficult to find. Such sentiments expressing gaps in PWP are consistent with what has been found in other specialties, indicating this may be a global issue.[8] [11] [12] [16] [17] Nonetheless, the current state of ophthalmology PWP complicates applicants' decision making and hampers their ability to determine which programs are a good fit. By maintaining an up-to-date, easy-to-use, and navigable program website, including key information, and utilizing social media tools, programs may increase their appeal to prospective residents by better aligning their PWP with applicants' interests.

We recognize the challenges programs face in developing their PWP. Time and resource constraints, institutional guidelines, preferences with respect to the publicity of particular program details, and other factors all impact PWP. Nonetheless, any efforts made to advance PWP may improve recruitment efforts.

There are certainly other important elements considered by applicants in the recruitment process, including program location, program reputation, the interview experience, etc. Though these are not thoroughly considered here, the purpose of this data is to provide an initial understanding of the applicants' perspective of the evolving impact of PWP on recruitment. The data presented may serve as a starting point for programs as they hone their own online identity.

Limitations of this study include the response rate as well as the group surveyed, given all responses came from medical students who applied to one ophthalmology program. Though the total number of respondents encompasses approximately two-thirds of the general applicant pool, there may be subgroups that were not surveyed. Furthermore, surveys are subject to self-reporting biases and may not always reflect true behavior. This survey was administered during the application period, near rank list submission, to minimize recall bias. However, the timing of each individual's response when compared with their interviews, application and rank list decisions, may have played a role in their answer selections.

Conclusion

In conclusion, residency programs' online efforts do impact recruitment to some extent. Programs should continue to refine their use of online tools as they seek to enhance the process of attracting applicants who will fit well in their respective programs.

Conflict of Interest

None declared.

Acknowledgments

The authors thank Erik Lehman, MS, for providing statistical consultation.

IRB Approval

The Pennsylvania State University Institutional Review Board approved this study.

Presentation

Oral presentation at 2016 Association of University Professors of Ophthalmology (AUPO), Program Coordinator's Annual Meeting, January 28, 2016, Ft. Lauderdale, Florida. Presenter: Mark Goerlitz-Jessen.

Supplementary Material

Applicant survey tool used in this study.

• References

• 1 Pew Research Center. Generations. Available at: http://www.pewinternet.org/2010/12/16/generations-2010/ . Accessed September 10, 2015
• 2 Pew Research Center. Social Media Usage: 2005–2015. Available at: http://www.pewinternet.org/2015/10/08/social-networking-usage-2005-2015/ . Accessed September 10, 2015
• 3 Brazin LR. Sources of information on postgraduate medical training programs – 2002 update. Med Ref Serv Q 2002; 21 (02) 1-14
• 4 Delzell Jr JE, Weick R, Weick M. How do medical students gather information about residency training programs?. Mo Med 2003; 100 (02) 153-154
• 5 Winters RC, Hendey GW. Do web sites catch residency applicants?. Acad Emerg Med 1999; 6 (09) 968-972
• 6 Deloney LA, Perrot LJ, Lensing SY, Jambhekar K. Radiology resident recruitment: A study of the impact of web-based information and interview day activities. Acad Radiol 2014; 21 (07) 931-937
• 7 Schweitzer J, Hannan A, Coren J. The role of social networking web sites in influencing residency decisions. J Am Osteopath Assoc 2012; 112 (10) 673-679
• 8 Mahler SA, Wagner MJ, Church A, Sokolosky M, Cline DM. Importance of residency program web sites to emergency medicine applicants. J Emerg Med 2009; 36 (01) 83-88
• 9 Kumar A, Sigal Y, Wilson E. Web sites and pediatric residency training programs in the United States. Clin Pediatr (Phila) 2008; 47 (01) 21-24
• 10 Mayo GL, Lindhorst GC, Rosende C. American ophthalmology graduate medical education and the web: current state of internet resource utilization. Am J Ophthalmol 2003; 135 (05) 708-709
• 11 Ashack KA, Burton KA, Soh JM. , et al. Evaluating dermatology residency program websites. Dermatol Online J 2016; 22 (03) pii :13030/qt7rx3j2dn
• 12 Reilly EF, Leibrandt TJ, Zonno AJ, Simpson MC, Morris JB. General surgery residency program websites: usefulness and usability for resident applicants. Curr Surg 2004; 61 (02) 236-240
• 13 Yousuf SJ, Kwagyan J, Jones LS. Applicants' choice of an ophthalmology residency program. Ophthalmology 2013; 120 (02) 423-427
• 14 Ebrahim S, Bowling A. Handbook of Health Research Methods: Investigation, Measurement and Analysis. New York, NY: McGraw-Hill International; 2005
• 15 San Francisco Match, Residency and Fellowship Matching Services. 2016 and 2017 Statistics. Available at: https://www.sfmatch.org/SpecialtyInsideAll.aspx?id=6&typ=2&name=Ophthalmology# . Accessed February 2, 2017
• 16 Silvestre J, Tomlinson-Hansen S, Fosnot J, Taylor JA. Plastic surgery residency websites: a critical analysis of accessibility and content. Ann Plast Surg 2014; 72 (03) 265-269
• 17 Embi PJ, Desai S, Cooney TG. Use and utility of Web-based residency program information: a survey of residency applicants. J Med Internet Res 2003; 5 (03) e22