Keywords
medical education - ophthalmology residency - residency match - virtual interviews
On May 8, 2020, the Program Director Council (PDC) and the Medical Student Education
Councils (MSEC) of the Association for University Professors of Ophthalmology (AUPO)
in conjunction with the San Francisco Match and AUPO Board of Trustees released recommendations
for the 2021 Ophthalmology Residency Application Cycle.[1] These recommendations were published in response to the changing national landscape
brought about by the novel coronavirus disease 2019 (COVID-19) pandemic and its anticipated
impact on the residency application and interview cycle. These recommendations addressed
away rotations, research, community service/volunteering, United States Medical Licensing
Examination (USMLE) step 2, the medical student performance evaluation (MSPE), letters
of recommendation, and the general application timeline. Included in these recommendations
was guidance urging residency programs to conduct virtual interviews (VIs) instead
in-person interviews. The VI format offers new challenges and opportunities for both
residency programs and applicants.
In a 2013 survey, Yousuf et al reported different factors that influenced how applicants
ordered their rank lists when applying to ophthalmology residency programs.[2] One of the highest ranked factors was applicants' perception of the relationship
between faculty members and residents. Factors associated with lower rankings included
perception of low resident morale and poor interview experience. These findings were
predicated on the ability of applicant's interaction with faculty and residents face-to-face.
Under the new VI format, opportunities for this type of engagement would theoretically
be limited. Still, the VI is the applicant's opportunity to share their story, personality,
and qualifications. With the change in format, dependence on technology and limitations
of nonverbal cues, the virtual platform requires a level of comfort with new variables.
A multidisciplinary task force at the University of California San Francisco found
that the following were key areas to address to allow for evidence based best practices:
adopting standardized interview questions, consciously addressing implicit bias, providing
practice opportunities to trainees, hosting social events, and collecting data for
assessment of the virtual process for further exploration.[3] This dialogue in other specialties led a group of residency program directors in
pediatrics to identify several specific factors to address when preparing for the
official interview day. These include correcting issues with technology that prohibit
clear and smooth communication, providing phone numbers to use as backup in case technical
difficulties arise, and addressing challenges with lighting and camera angle. These
factors led researchers to suggest recording practice sessions to test these variables
that are unique to the virtual setting.”[4] To give prospective applicants, in this year's application cycle, a source of familiarity
with these considerations in mind, we created a medical student–led mock VI initiative
for the 2020 to 2021 cycle of ophthalmology residency applicants and conducted a pre-
and post-survey of participants to assess the perceived effectiveness of these sessions
in improving applicants' comfort with the VI format.
Methods
Mock Virtual Interviews
This study adhered to the tenets of the Declaration of Helsinki. The University of
Miami Institutional Review Board (IRB) waived the need for IRB approval. Mock VI practice
opportunities were developed by the authors (B.A., V.L., A.P., A.W., S.A., and .A.C).
These mock VI sessions were free, student led, nonsponsored, and not affiliated with
specific institutions. Announcements were marketed and disseminated via social media
platforms including Twitter and Instagram starting on August 2, 2020. A Google document
requesting name, phone number, e-mail, and medical school of each voluntary participant
was made available on August 22, 2020, and announced on the previously listed forums
along with other additional forums. These additional forums included the Honestly
Bilal podcast, the “OphthoMatch 2020–2021” applicants' discussion Google sheets document
(Alphabet Inc., Mountain View, CA), and the associated anonymous discussion channel
for all applicants on the web application Discord (San Francisco, CA). Sign-ups were
closed once a maximum of 60 participants had been reached.
Once the sign-up was closed, the mock interviewers internally coordinated availability
via the web site, namely, SignUp Genius (SignUp Genius, Charlotte, NC). Once finalized,
a separate SignUp Genius link was created and distributed via e-mail on September
28, 2020, to participants who then picked a date and time of their choice between
October 12, 2020, and November 2, 2020. Using the Zoom (Zoom Video Communications,
San Jose, CA) video-telecommunications platform, meetings were created, scheduled,
and distributed to each interviewer and interviewee under a single account. To limit
external bias and to promote a student-oriented collaborative culture of this event,
no application materials or USMLE/COMLEX: Comprehensive Osteopathic Medical Licensing
Examination of the United States scores were required or requested, and only the 6
originating student members of the initiative (BA, VL, AP, AW, SA, AC) assumed roles
as mock interviewers.
Immediately before each 20-minute mock VI, each interviewee completed the preinterview
survey which was distributed as a Google form (Appendix). All interviewees were given the option to have their session recorded for viewing
at their own convenience once the call ended. Interviewers selected questions from
a list of 19 questions from the article “The Interview Trail: How to Succeed on your
Residency/Fellowship Interview,” written by J.S. (
http://www.retinapodcast.com/equal-round-and-reactive/2017/10/20/the-interview-trail-how-to-succeed-on-your-residencyfellowship-interview
). At the end of each VI, each interviewee was then sent the postinterview survey,
another Google form (Appendix) which included identical questions and answer choices like preinterview survey.
Interviewees were also sent the recording of the interview if they opted in to be
recorded. A follow-up survey was distributed via e-mail to all 35 participants after
the match starting on February 18, 2021. This Google form was closed on February 25,
2021.
Statistical Analyses
Descriptive statistics and data analyses were performed using IBM SPSS Statistics,
version 27 (IBM Corp., Armonk, NY). The data were not normally distributed, so nonparametric
tests were performed. The Kruskal–Wallis test, the nonparametric counterpart of one-way
analysis of variance (ANOVA), was performed to determine if there was a difference
in survey responses between groups divided by gender, ethnicity, region, and level
of postgraduate education. The Wilcoxon's signed-rank test for hypothesis testing
of repeated measurements on a single sample was used to evaluate whether or not the
survey responses before and after the mock interview were different. All tests were
two tailed, and test significance was determined at the α level of 0.05.
Results
Of the initial 60 participants interested who filled out the initial sign-up document,
41 followed up via the SignUp Genius form. Out of the 41 participants scheduled, 35
participants attended their respective sessions and completed both surveys, and 23
participants opted to record their session. In total, participants represented 27
allopathic medical schools, 2 osteopathic medical schools, and 5 international medical
schools. When considering the demographic data of the participants ([Table 1]), there were no statistically significant differences in survey responses between
different genders, races/ethnicities, medical school locations, and level of postgraduate
education. The average scores of the participants' pre- and postinterview survey responses
on a 5-point Likert's scale are summarized in [Table 2].
Table 1
Demographic information of the virtual interview participants
Characteristic
|
Number (%)
|
Gender
|
• Female
|
18 (51)
|
• Male
|
17 (49)
|
Race/Ethnicity
|
• White
|
19 (54)
|
• Asian/Pacific Islander
|
13 (37)
|
• Black or African American
|
1 (3)
|
• Hispanic or Latino
|
1 (3)
|
• Other
|
1 (3)
|
Medical school location by U.S. region
|
• Northeast
|
5 (14)
|
• Midwest
|
10 (29)
|
• South
|
8 (23)
|
• West
|
7 (20)
|
• International
|
5 (14)
|
Postgraduate degrees
|
• Masters of science
|
5 (14)
|
• PhD
|
1 (3)
|
Total number of participants
|
35 (100)
|
Table 2
Descriptive statistics of pre- and post-interview survey responses
Survey question
|
Preinterview (mean ± SD)
|
Postinterview (mean ± SD)
|
p-Value[a]
|
How prepared do you feel for virtual interviews with residency programs?
|
2.9 ± 1.0
|
3.6 ± 0.9
|
0.0003[b]
|
Mean difference in score (95% CI)
|
+0.66 (0.15–0.96)
|
|
How likely are you to practice virtual interviews with someone you know?
|
4.4 ± 0.7
|
4.7 ± 0.5
|
0.0023[b]
|
Mean difference in score (95% CI)
|
+0.31 (0.13–0.50)
|
|
How likely are you to practice virtual interviews?
|
4.4 ± 0.5
|
4.8 ± 0.4
|
0.20
|
Mean difference in score (95% CI)
|
+0.11 (−0.07 to 0.30)
|
|
How likely are you to practice virtual interviews in the same room/area as you will
during the official interview season?
|
4.3 ± 1.1
|
4.7 ± 0.6
|
0.07
|
Mean difference in score (95% CI)
|
+0.37 (−0.03 to 0.77)
|
|
Abbreviations: CI, confidence interval; SD, standard deviation.
Note: Likert's scale: 1 = very unprepared/unlikely, 2 = somewhat unprepared/unlikely,
3 = neutral/undecided, 4 = somewhat prepared/likely, 5 = very prepared/likely.
a Wilcoxon's signed-rank test.
b Indicates statistically significant difference (p < 0.05).
Participants felt significantly more prepared for virtual residency interviews after
the mock interview (mean rank = 164.5) than before the mock interview (mean rank = 6.5;
Z = − 3.57, p = 0.0003). The median Likert-scaled scores were 3 for the preinterview and 4 for
the postinterview surveys. Prior to the mock interview, only 12 participants had responded
“Somewhat prepared” to the question, “How prepared do you feel for virtual interviews
with residency programs?.” In the postinterview survey, most participants (22/35)
responded “Somewhat prepared” ([Fig. 1A]).
Fig. 1 (A) Mock virtual interview participant self-reported preparedness pre and post mock
virtual interviews. (B) Mock virtual interview participant self-reported willingness to practice interviews
pre and post mock virtual interviews.
Participants were significantly more likely to practice VIs with someone they knew
after the mock interview (mean rank = 84) than before the mock interview (mean rank = 7;
Z = − 3.05, p = 0.0023). The median Likert-scaled scores of the survey question “How likely are
you to practice virtual interviews with someone you know?” were 4 for the preinterview
and 5 for the postinterview surveys. All of the participants who responded “Neutral/Undecided”
in the preinterview survey (n = 4) changed their response to “Somewhat Likely” after the interview. In the postinterview
survey, the majority of participants (24/35) responded “Very Likely” ([Fig. 1B]).
There were no statistically significant pre- and postinterview differences in the
responses to the question “How likely are you to practice virtual interviews?” (p = 0.20). Most participants responded “Very Likely” to “How likely are you to practice
virtual interviews?.” Most participants also responded “Very Likely” to “How likely
are you to practice virtual interviews in the same room/area as you will during the
official interview season?” ([Fig. 1B]). Of the five participants who responded in the preinterview survey “Very Unlikely,”
“Somewhat Unlikely,” or “Neutral,” only one participant did not change their answer
to “Very Likely” after the mock interview. However, there were no statistically significant
differences in the pre- and postinterview responses (p = 0.07).
Of the 35 participants who attended the VI mock sessions, 20 completed the follow-up
survey. There were an equal number of participants who responded either “Helped Somewhat”
(n = 9) or “Helped Greatly” (n = 9) to “How much did the VI mock exercise help you for the actual interview season?”
in the follow-up survey. There were two participants who responded to the same question
with “Helped Very Little.” The majority of follow-up survey respondents (17/20) reported
that they had additional practice in the virtual environment for interviews after
the VI mock exercise and a majority (n = 13, 65%) practiced once to five times ([Fig. 2]). The participants of the follow-up survey either did not have a preference for
mock interviewers (n = 10) or preferred to practice with faculty or current trainees (n = 10). Most participants (15/20) felt that not sharing their application materials
added to a collegial practice setting.
Fig. 2 Mock virtual interview participant self-reported number of virtual interview practice
sessions versus final ophthalmology residency match status.
Most follow-up survey respondents (15/20) matched into an ophthalmology residency
program. There were three participants who did not match, and two participants who
did not report their match status. Of the 15 participants who reported matching, 67%
matched into a program that was one of their top three choices on their rank order
list ([Fig. 3]).
Fig. 3 Self-reported final ophthalmology residency match result position on individual submitted
rank list for mock virtual interview participants.
A Mann–Whitney U-test showed that there was no difference in how helpful the VI mock exercise was
during the actual interview season between matched and unmatched participants. There
was a strong positive relationship (r = 0.573, p = 0.026) between how helpful the VI mock exercise was during the actual interview
season and the number choice of the ophthalmology residency program that participants
matched on their rank list. This relationship suggested participants who matched at
programs lower on their rank list (i.e., fifth or sixth choice) were likely to find
the VI mock exercise more helpful during the actual interview season than participants
that matched at programs higher on their rank list (i.e., first or second choice).
There was no association between how helpful the VI mock exercise was perceived to
be during the actual interview season and how many times the participants practiced
virtual interviewing. A Mann–Whitney U-test showed that there was no difference between the matched and unmatched participants
in the number of times they practiced virtual interviewing after their VI mock exercise.
There was also no association between the number of times participants practiced virtual
interviewing and the number of choice of the program they matched on their rank lists.
Discussion
This study illustrates the perceived impact of more formalized preparation for VIs
on students applying for ophthalmology residency in the 2020 to 2021 match. The data
from our study indicate that from a small sample pool of applicants, nationally and
internationally, students felt more prepared after a mock VI exercise. Additionally,
on completion of our mock VI exercise, participants were more likely to be inclined
to practice VIs with someone they already knew.
All specialties under the recommendation of the Association of American Medical Colleges,
along with ophthalmology under the AUPO, subscribed to the VI format, this residency
application season is to mitigate the risk of COVID-19. Medical students and residency
programs across the country adapted to this alternative format, but navigating the
nuances of the virtual platform may be challenging, and little is known about how
applicants perceived the benefits of mock VIs.
There are many technical variables that are unique to VIs, such as maintaining eye
contact with a camera, a relative lack of visual cues and body language, adjusting
lighting/background, and optimizing audiovisual and internet technology.[5]
[6] Practicing mock interviews with the intention to develop comfort with these variables
may allow students to feel more prepared and confident. Previous studies have indicated
that mock interviews in a nonvirtual format that boosts interviewees' confidence and
ultimately improves performance on interview day.[7]
The interview is an important element of the ophthalmology residency application.
Previous studies have shown that the majority of program directors in many medical
specialties place the greatest weight on the interview when reviewing applicants.[7] More specific to ophthalmology, a previous study of ophthalmology residency selection
committees, who were surveyed, suggested that interviews were the best predictor of
future resident performance.[8] However, practice and personal assessment may contribute to improved match outcomes
by increasing the odds of superior interview performances. This may be especially
true for candidates not considered to be as strong in other areas of their application
such as test scores, medical school ranking, or honor society status. In the current
study, we showed that applicants who underwent mock VIs felt more prepared for their
upcoming actual VIs. Our follow-up survey showed that a majority of these participants
who responded were successful in matching.
The virtual format may offer several other benefits in addition to mitigating viral
infection risk. VIs offer cost and time advantages, along with the potential to utilize
a hybrid approach of screening applicants virtually before an onsite interview.[9]
[10] VIs may also facilitate the very time-sensitive Supplemental Offer and Acceptance
Program (SOAP) process with unmatched applicants and unfilled programs.[11] Nonetheless, time will tell if the potential merits of this format are strong enough
to replace or supplement the benefits of traditional formats in years to come.
Limitations
This study has several limitations, including the small sample size. The format of
the mock VIs in this study was conducted in a manner where no applicant material was
necessary to be eligible to participate to increase participation and limit fear of
judgement by fellow peers. With the lack of information about a participant's board
scores, research, clinical grades, leadership, and activities, the interviewers were
unable to simulate potential questions surrounding those topics. This was intentional
for this cohort to maintain confidentiality given that interviewees and interviewers
were all medical students applying for ophthalmology residencies in the same year.
However, there may be benefits to interviewing without other application materials.
One study of surgical residency candidates showed that knowledge of the complete application
of candidates contributed to interviewer bias.[12] Another limitation of the current study was the lack of formal training and standardization
of the interviewers. Participants overall, who responded to our follow-up survey,
found the VI mock exercise helpful for the actual interview setting and continued
practicing in the virtual setting. Of those who responded to our follow up survey,
with regard to where they matched on their rank list, 67% matched into one of their
top three ranked programs. It is noteworthy that there is a positive correlation between
matching lower on one's rank list and finding the mock VI exercise helpful; however,
the sample size was extremely small and there are multiple potential confounding factors.
For example, applicants with less access to practice resources at their home institution
may also be less “competitive” in residency applications due to factors such as access
to letters of recommendation, prestige of school, and access to scholarly projects.
Conclusion
The COVID-19 pandemic has created incredible uncertainty for medical students applying
to residency programs throughout the United States, including the new VI format. Our
study demonstrated potential benefits of mock VI preparation for residency applicants.
As we gain more information from this year's application cycle, future studies will
further clarify how best VIs can be conducted, and how applicants can optimally be
prepared for the residency interview.
Preinterview Survey and Postinterview Survey
-
Gender
-
Male
-
Female
-
Nonbinary
-
Prefer not to say
-
Other
-
If you answered “Other”, please type in your gender identity:
-
Please specify your ethnicity
-
What region in the United States do you attend medical school?
-
Northeast (CT, ME, MA, NH, RI, VT, NJ, NY, PA)
-
Midwest (IL, IN, MI, OH, WY, IA, KS, MN, MO, NE, ND, SD)
-
South (DE, DC, FL, GA, MD, NC, SC, VA, WV, AL, KY, MS, TN, AR, LA, OK, TX)
-
West (AZ, CO, ID, NM, MT, UT, NV, WY, AK, CA, HI, OR, WA)
-
I attend medical school in another country.
-
Have you earned a postgraduate degree? (MSc, MBA, MA, MFA, MEng, Ph.D., etc.)
-
If you answered yes above, what type of postgraduate degree have you earned?
-
How prepared do you feel for virtual interviews with residency programs?
-
Very unprepared
-
Somewhat unprepared
-
Neutral
-
Somewhat prepared
-
Very prepared
-
How likely are you to practice virtual interviews with someone you know?
-
Very unlikely
-
Somewhat unlikely
-
Neutral/undecided
-
Somewhat likely
-
Very likely
-
How likely are you to practice virtual interviews?
-
Very unlikely
-
Somewhat unlikely
-
Neutral/undecided
-
Somewhat likely
-
Very likely
-
How likely are you to practice virtual interviews in the same room/area as you will
during the official interview season?
-
Very unlikely
-
Somewhat unlikely
-
Neutral/undecided
-
Somewhat likely
-
Very likely
Follow-up Survey
-
How much did the VI mock exercise help you during the actual interview season?
-
It helped very little
-
It helped somewhat
-
It helped greatly
-
How many times did you practice in the virtual environment (i.e., With Zoom, Webex,
and others) after the VI mock exercise?
-
0 times
-
1–5 times
-
6–10 times
-
>10 times
-
Did you prefer medical students as the mock interviewers during the VI mock exercise?
-
I preferred practicing with faculty or current trainees
-
I did not have a preference
-
I preferred practicing with other applicants
-
Did you feel not sharing your application material (i.e., USMLE scores and research,
extracurricular) added to a nonjudgmental and collegial environment for the VI mock
exercise?
-
No, I preferred having my application viewed
-
I did not have a preference
-
Yes, I felt not sharing my application materials added to a collegial practice setting
-
Did you match into a residency in ophthalmology?
-
Which of the following reflects where you matched on your rank list?
-
First choice
-
Second choice
-
Third choice
-
Fourth choice
-
Fifth choice
-
Sixth choice
-
Seventh choice
-
Eighth choice
-
Ninth choice
-
Tenth choice
-
Eleventh choice
-
Twelfth choice