Keywords
COVID-19 - ophthalmology trainees - virtual interviews - medical education
During the height of the coronavirus disease 2019 pandemic, the introduction of virtual interviews was essential for ensuring the continuity and integrity of the fellowship interview process while observing health and safety protocols due to the public health emergency. Concerns for social distancing and restrictions on travel have since subsided, allowing many prepandemic practices to be reinstated. Still, virtual interviews remain the predominant format for residency and fellowship interviews despite limited understanding of how this platform affects the interview process.[1]
According to a 2021 survey study by Patel et al, applicants were generally satisfied with virtual interviews due to reduced costs and increased ability to attend a greater number of interviews.[1] Other studies surveying applicants, interviewers, and committee members within various specialties demonstrated more equivocal results.[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14] While participants on both sides of the interview process envision a future role for virtual interviews,[7]
[8]
[11]
[12]
[14] few believe they will completely replace in-person interviews.[4]
[9]
[12]
[13] Commonly cited shortcomings include difficulty assessing program fit,[1]
[2]
[3]
[5]
[6]
[13]
[14] difficulty implementing and conducting the virtual interview,[5]
[14] and ambiguity in how a virtual format would ultimately impact perceptions of the applicant.[6]
[9] Conversely, applicants reported many benefits, including reduced costs,[1]
[2]
[3]
[7]
[11]
[13]
[14] more favorable travel burden,[3]
[14] decreased time commitment,[11]
[13]
[14] improved stress,[7] lower carbon footprint,[7]
[14] and greater schedule flexibility.[11] Only two studies demonstrated overwhelmingly positive reviews, reporting applicants felt they were adequately represented, formed sufficient interpersonal connections with interviewers, and gained a strong understanding of program culture and fit.[8]
[11]
Despite mounting research on the virtual interview process, prior studies did not include applicants with in-person interviews, and therefore could not directly compare the two formats. Therefore, previous results are limited due to the lack of a standard control group. With the reintroduction of partial in-person interviews in the 2022 to 2023 application year, an opportunity for clear comparison is presented. The purpose of this study is to evaluate the relative benefits and drawbacks of in-person versus virtual interviews by surveying applicants who have attended both.
Methods
This study was approved by the Wills Eye Hospital Institutional Review Board (IRB) and the Bascom Palmer Eye Institute IRB. Program directors for six fellowship programs at both hospitals were contacted and asked to provide email information for their applicants from the 2022 to 2023 San Francisco Match fellowship application cycle. In total, 303 applicants were identified. The fellowship programs included cornea, ocular oncology, glaucoma, retina, academic global ophthalmology, and other. The oculoplastics fellowship did not conduct interviews this particular application cycle and were therefore excluded.
The IRB-approved Qualtrics survey was sent to Wills Eye Hospital applicants on January 23, 2023. Two reminder emails were sent on January 30, 2023 and February 06, 2023. The IRB-approved Qualtrics survey was sent to Bascom Palmer Eye Institute applicants on February 15, 2023. Two reminder emails were sent on February 20, 2023 and February 27, 2023. The survey closed on March 03, 2023. The survey included 19 questions regarding demographic information, the number of interviews that applicants applied to, were invited to, and interviewed at, monetary costs, perceived satisfaction and effectiveness of the virtual and in-person interview process, perceived strengths and limitations of the virtual interview, perceived strengths and limitations of the in-person interview, ideal interview length, and preferred interview format.
All data were analyzed using statistical software (IBM SPSS 25 Statistics, Armonk, NY). Statistical significance was considered to be a two-sided p-value < 0.05.
Results
Applicant Demographics
In total during the 2022 to 2023 application cycle, there were 507 registered ophthalmology fellowship applicants and 422 submitted rank lists.[15] A total of 311 applicants to the cornea, retina, glaucoma, ocular oncology, and academic global ophthalmology fellowship programs at Wills Eye Hospital and Bascom Palmer Eye Institute were identified. Of those, 59 applicants responded to our survey (overall response rate of 19%), with 20 applying to cornea (33.9%), 18 applying to retina (30.5%), 12 applying to glaucoma (20.3%), 8 applying to other (13.6%), and 1 applying to ocular oncology (1.7%).
The majority of applicants were between the ages of 20 and 35 (91.3%), did not identify as an underrepresented minority (86.4%), and were male (53.0%) ([Table 1]). Applicants from residency programs/affiliated institutions in the Northeast had the highest representation among all regions (44.1%), followed by the Southern U.S. (22.0%), Midwestern U.S. (15.3%), Western U.S. (10.2%), and outside the U.S. lower 48 states (8.5%) ([Table 1]).
Table 1
Survey demographics
|
Q2: Which fellowship program did you apply to during the 2022–2023 application cycle?
|
N (%)
|
|
Cornea
|
20 (33.9)
|
|
Glaucoma
|
12 (20.3)
|
|
Ocular oncology
|
1 (1.7)
|
|
Retina
|
18 (30.5)
|
|
Other
|
8 (13.6)
|
|
Q3: What is your current age?
|
N
(%)
|
|
< 20–30 y
|
31 (53.4)
|
|
31–35 y
|
22 (37.9)
|
|
36–40 y
|
4 (6.9)
|
|
41–> 45 y
|
1 (1.7)
|
|
Q4: What is your gender?
|
N
(%)
|
|
Female
|
26 (44.1)
|
|
Male
|
31 (53.0)
|
|
Prefer not to say
|
2 (3.4)
|
|
Q5: On the SF Match application, did you mark yourself as an underrepresented minority?
|
N
(%)
|
|
Yes
|
6 (10.2)
|
|
No
|
51 (86.4)
|
|
Prefer not to say
|
2 (3.4)
|
|
Q6: Where is your residency program/affiliated institution located?
|
N
(%)
|
|
Northeast U.S.
|
26 (44.1)
|
|
Midwest U.S.
|
9 (15.3)
|
|
Southern U.S.
|
13 (22.0)
|
|
Western U.S.
|
6 (10.2)
|
|
Outside the U.S. lower 48 states
|
5 (8.5)
|
|
Q7: During the 2022–2023 fellowship match cycle, how many TOTAL programs did you apply to?
|
N
(%)
|
|
< 5
|
2 (3.4)
|
|
5–10
|
5 (8.5)
|
|
10–15
|
11 (18.6)
|
|
15–20
|
13 (22.0)
|
|
> 20
|
28 (47.5)
|
Note: This demonstrates the distribution of all survey responses regarding fellowship subspecialties, age, gender, status as an underrepresented minority, geographical region of their residency program, and total number of programs applied.
Application Cycle Statistics
Out of 59 survey respondents, 47.5% applied to more than 20 fellowship programs ([Table 1]). Most applicants were invited to and attended between 15 and 19 total interviews (30.5% and 30.5%, respectively). More specifically, most were invited to 5 to 9 virtual interviews (37.9%) and 5 to 9 in-person interviews (36.8%), but subsequently attended 5 to 9 virtual interviews (39.3%) and less than 5 in-person interviews (41.1%). There was no statistically significant difference between the number of virtual and in-person interviews received (p = 0.07) or attended (p = 0.24). A more complete representation of the data can be seen in [Figs. 1] and [2].
Fig. 1 During the 2022 -2023 fellowship match cycle, how many interviews were you invited to?
Fig. 2 During the 2022 -2023 fellowship match cycle, how many interviews did you attend?
The overall cost of virtual interviews was between $0 and $250 for 94.6% of applicants, significantly less than the $1,000 to $5,000 overall cost of in-person fellowship interviews reported by the majority of applicants (52.7%; p < 0.001) ([Fig. 3]).
Fig. 3 During the 2022 -2023 fellowship match cycle, what was the monetary cost for you to attend interviews?
Applicant Feedback and Suggestions
The average satisfaction of virtual interviews on a scale from 0 (not satisfied) to 10 (very satisfied) was 6.9 ± 2.1, while the average satisfaction of in-person interviews was 7.3 ± 2.7 (p = 0.36; [Table 2]). The average perceived effectiveness of virtual interviews on a scale from 0 (not effective) to 10 (very effective) was 6.3 ± 1.9, while the average perceived effectiveness of in-person interviews was 8.0 ± 1.5 (p < 0.001; [Table 2]). When asked which format they prefer, 28.8% of applicants picked virtual interviews, 39.0% of applicants picked in-person interviews, and 32.2% of applicants picked the option to choose either (p = 0.62; [Table 2]).
Table 2
Overall survey results
|
Virtual versus in-person interview satisfaction ratings (0 = least satisfied, 10 = most satisfied)
|
Mean ± SD
|
p-Value
|
|
The virtual interview process
|
6.9 ± 2.1
|
0.36
|
|
The in-person interview process
|
7.3 ± 2.7
|
|
Virtual versus in-person interview effectiveness ratings (0 = least effective, 10 = most effective)
|
Mean ± SD
|
p
-Value
|
|
The virtual interview process
|
6.3 ± 1.9
|
< 0.001*
|
|
The in-person interview process
|
8.0 ± 1.5
|
|
Virtual versus in-person interview preference
|
N
(%)
|
p
-Value
|
|
In-person interview
|
23 (39.0)
|
0.62
|
|
Virtual interview
|
17 (28.8)
|
|
Option to choose either
|
19 (32.2)
|
Abbreviation: SD, standard deviation, (*) indicates significance.
Note: Satisfaction and effectiveness ratings were given on a scale of 0 to 10, where 0 was the lowest and 10 was the highest score. Interview preferences were recorded as a total number of applicants who selected one of the three options. p-Values for the satisfaction and effectiveness ratings were obtained via t-test, while interview preference was analyzed using a chi-squared distribution.
Reponses from fellowship applicants in the five different regions differed significantly when rating their satisfaction with virtual interviews (p = 0.025), as well as when comparing the difference between their satisfaction and effectiveness ratings for virtual and in-person interviews (p = 0.006 and p = 0.05, respectively) ([Table 3]). Applicants from outside the U.S. lower 48 states were on average 2.0 points more satisfied with virtual interviews than the next highest region. They also rated virtual interviews 0.4 points more effective than in-person on average, compared with all other regions who rated in-person more effective. However, there were no differences in the distribution of their preferences for future application cycles (p = 0.15). There were also no significant differences in the responses of applicants who identified as male compared to female, nor those who identified as underrepresented minorities compared with those who did not.
Table 3
Survey results by region
|
Virtual versus in-person interview satisfaction ratings (0 = least satisfied, 10 = most satisfied)
|
|
Region
|
Mean virtual interview rating (SD)
|
Mean in-person interview rating (SD)
|
Difference in means
(SD)
|
|
Northeast U.S. (n = 23)
|
6.27 (2.1)
|
7.91 (1.7)
|
1.91 (3.1)
|
|
Midwest U.S. (n = 9)
|
6.56 (1.3)
|
8.00 (2.1)
|
1.44 (2.4)
|
|
Southern U.S. (n = 12)
|
7.00 (2.0)
|
6.67 (2.9)
|
–0.33 (3.2)
|
|
Western U.S. (n = 6)
|
7.50 (2.2)
|
6.67 (3.4)
|
–0.83 (4.6)
|
|
Outside the U.S. lower 48 states (n = 5)
|
9.50 (0.9)
|
5.00 (4.8)
|
–4.40 (5.3)
|
|
ANOVA results
|
p
= 0.025*
|
p = 0.154
|
p
= 0.006*
|
|
Virtual versus in-person interview effectiveness ratings (0 = least effective, 10 = most effective)
|
|
Region
|
Mean virtual interview rating (SD)
|
Mean in-person interview rating (SD)
|
Difference in means (SD)
|
|
Northeast U.S. (n = 23)
|
6.04 (1.9)
|
8.31 (1.2)
|
2.27 (2.0)
|
|
Midwest U.S. (n = 9)
|
6.00 (1.9)
|
7.89 (1.05)
|
2.56 (1.9)
|
|
Southern U.S. (n = 12)
|
5.85 (1.8)
|
7.89 (1.9)
|
1.85 (2.5)
|
|
Western U.S. (n = 6)
|
7.17 (2.1)
|
7.33 (1.9)
|
0.17 (2.8)
|
|
Outside the U.S. lower 48 states (n = 5)
|
8.40 (1.1)
|
8.00 (1.7)
|
-0.40 (2.3)
|
|
ANOVA results
|
p = 0.586
|
p = 0.572
|
p
= 0.050*
|
Abbreviation: SD, standard deviation, (*) indicates significance.
Note: Satisfaction and effectiveness ratings were given on a scale of 0 to 10, where 0 was the lowest and 10 was the highest score. All p-values were obtained via analysis of variance (ANOVA).
When ranking the strengths and limitations of the virtual and in-person interview processes, the highest ranked limitations of the virtual interview process based on the median rankings were: limited exposure to details of the program structure (i.e., call schedule, rotation blocks), limited opportunity to exhibit applicants' strengths to the program, and limited exposure to the fellows. Conversely, the highest ranked strengths based on the median rankings were less pressure during interviews, greater scheduling flexibility, ability to interview at more fellowship programs, and shorter interview schedule. The highest ranked limitations based on the median rankings of the in-person interview process were more pressure during interviews, inability to interview at all desired fellowship programs, and decreased scheduling flexibility and the highest ranked strengths based on median rankings were greater exposure to details of the program structure (i.e., call schedule, rotation blocks), greater ability to exhibit an applicant's strengths to the program, and greater exposure to the geographic location/city.
For both virtual and in-person interviews, the majority of applicants believe that a length of 15 or 20 minutes is optimal (virtual 83%, in-person 76%). However, when comparing answers of individual applicants, respondents preferred longer in-person interview duration than virtual interviews (18.6% vs. 6.8%, p=0.049) ([Fig. 4]).
Fig. 4 What would be the optimal length of time for each interview type?
Discussion
Overall, this study demonstrated high satisfaction rates for both virtual and in-person interview formats. Applicants did, however, perceive in-person interviews to be significantly more effective than virtual ones. Interestingly, a third of the applicants preferred that programs present the option to choose either in-person or virtual interviews, but the majority (39%) felt it appropriate to transition away from virtual interviews altogether.
These results are supported by the overall consensus in the literature. Survey respondents on both sides of the interview process and from an array of different specialties believe virtual interviews will continue to have a role in the process.[6]
[8]
[10]
[11]
[13] However, few believe they will continue to completely supplant in-person interviews.[3]
[8]
[11]
[12]
[14] Although it is unclear how programs will choose to structure their interviews in the future, there could be an opportunity to offer both or a new hybrid format.
Beyond applicant preferences, virtual interviews may have drawbacks for fellowship programs. Virtual interviews can negatively impact an applicant's ability to compare programs, fundamentally altering their decision-making process. Our study shows that 16% of the fellowship applicant cohort believes that limited exposure to details of the program structure (e.g., call schedule, rotation blocks) is one of the greatest limitations of the virtual interview. This is further supported by the literature, where difficulty assessing program fit is the most commonly reported drawback of the virtual interview.[1]
[2]
[4]
[6]
[12]
[13]
[14]
Meanwhile, our study demonstrated two major strengths of in-person interviews. The largest strength was increased exposure to the details of the program structure, a finding which mirrored the largest weakness of virtual formats. The other commonly cited benefit was a greater ability to exhibit an applicant's strengths, a sentiment that may also benefit programs by allowing them to identify the most qualified and suitable candidates for their positions.
Despite many reported benefits of virtual interviews, including reduced travel burden, carbon footprint, and costs, and greater schedule flexibility, this format may not confer meaningful changes to an applicant's interview experience.[1]
[2]
[6]
[10]
[12]
[13] Our study found that although there were significant cost and schedule flexibility discrepancies reported between virtual and in-person interviews, there was no significant difference in the actual number of virtual versus in-person interviews attended by applicants. However, these results may be affected by an overwhelming representation of Northeast and Midwestern applicants. Applicants further from the East Coast, especially those from outside the mainland United States, see a greater value in maintaining virtual interviews, finding them both more satisfactory and effective than other regions. Although the current study suggests that cost may not be the driving factor influencing interview attendance, further research incorporating a wider range of regional demographics may be needed to better understand the factors that drive interview attendance among applicants.
A few limitations may have affected the results of this study. Given the inherent nature of a survey study, a primary limitation is the possibility of response, nonresponse, and sampling biases. Additionally, many programs offer in-person second-look opportunities after the completion of interviews, which could compensate for the limited exposure to a program's culture during virtual interviews. If this pairing were to be considered equally effective by applicants, a cost–benefit analysis may help elucidate the future role of virtual interviews combined with an in-person second look. Finally, the small sample of underrepresented minority applicants may also limit our study's generalizability. Failure to address increased economic and social pressures may understate the negative impacts of costs and flexibility in our study.
Although results from the study did not reveal a statistically significant, “most limiting” flaw or benefit with either interview format, the reported benefits of in-person interviews and weakness of virtual ones complement one another. When considering both options, applicants were confident that they could exhibit their strengths and better understand a program's structure and culture when physically present for interview days. Although these parallels might present a juxtaposition, the alignment may also serve to highlight the biggest strengths and weaknesses of each format.
