Keywords USMLE - OKAP - written qualifying exam - resident - board
Medical training programs have a variety of methods for evaluating applicants and residents, including subjective, objective, and standardized systems. However, it can often be difficult to foresee which applicants will be successful in a residency program and which residents will be successful physicians. To help academic institutions measure resident progress as physicians in training, surrogate end points are often used, such as the Ophthalmology Knowledge Assessment Program (OKAP) and board certification which are designed to measure fund of knowledge and clinical reasoning. One study has demonstrated that clinical performance, as judged by faculty evaluations, does not correlate with OKAP performance, which suggests that there is more to becoming a successful physician than acquiring fund of knowledge.[1 ] Board certification is an important aspect as it has become a standard part of the licensing process and determining privileges to practice at institutions. There have been a few recent studies published investigating correlations between standardized evaluations such as the United States Medical Licensing Exam (USMLE) Step 1, Ophthalmology Knowledge Assessment Program (OKAP) and American Board of Ophthalmology Written Qualifying Examination (ABO-WQE).[2 ]
[3 ]
[4 ]
[5 ] Some of these studies have suggested a correlation between performance on the OKAP examinations and passing the ABO-WQE, which seemed contrary to the experience at our institution. Therefore, we have set out to perform an analysis in similar fashion to these prior studies among the graduating residents at our institution.
Methods
Data were collected from the records of the University of South Florida Residency Program in Ophthalmology from the graduating classes between 1999 and 2011. Each class consisted of three or four members. Forty-one resident physicians were identified; no residents were entirely excluded from the study. The following data were obtained: the three-digit score from USMLE Step 1 along with the national mean and standard deviation as reported by the Electronic Residency Application Service; OKAP examination scores for each of the 3 years of residency training; and the ABO-WQE scores up to three attempts and national passing cutoff. USMLE Step 1 scores for 10 residents could not be located; these residents were omitted from any analysis involving the USMLE Step 1 but were still utilized in analysis of the OKAP examination scores from year to year and to the ABO-WQE first attempt. OKAP examinations scores from all 3 years of training were available for all 41 residents. ABO-WQE first attempt scores were available for 40 of the 41 participants; the 1 participant never attempted to take the ABO-WQE and this participant was included in the analysis of the ABO-WQE as “fail.” USMLE Step 2 and Step 3 scores were not included as these results are rarely available to residency programs during the application process.
The three-digit score from the USMLE Step 1 was converted into a percentile using the national mean and standard deviation, thus allowing for a standard comparison across years that also served as a rank among the population. We also used a result of 30th percentile on the OKAP examination as passing which is close to the mean passing cutoff for the ABO-WQE first attempt between 1999 and 2011.[3 ] In addition, we analyzed ABO-WQE pass rates at various OKAP performance levels from starting at the 10th percentile and increasing by 10 percentile points.
Univariable analyses including the chi-square test and Fisher exact test were used to examine the relationship between the passing rates on the USMLE Step 1, OKAP examinations, and ABO-WQE first attempt.
The Student t -test was used to compare the mean percentiles scores on the USMLE Step 1, OKAP year 1 examination, OKAP year 2 examination, and OKAP year 3 examination between residents who failed and passed the first attempt on the ABO-WQE.
Pearson rank correlations were used to evaluate the relationship between percentile scores on the USMLE Step 1, OKAP examinations, and ABO-WQE first attempt, which provided a ranking in the population of all residents rather than a ranking in the sample of residents from our institution. In addition, using the Student t -test, we compared the ABO-WQE first attempt passing rates between residents who never passed any of the OKAP examinations and residents who passed at least one OKAP examination, and also between residents who passed all the OKAP examinations and residents who did not pass at least one OKAP examination.
Results
[Table 1 ] presents the sample sizes of the variables used in analysis as well as mean, standard deviation, and range. The pass rate on the ABO-WQE on first attempt was 78 and 87.8% within the first 3 years. The Pearson correlations among the percentile scores for the USMLE Step 1, OKAP examinations, and the ABO-WQE results are listed in [Table 2 ]. Performance, as measured by percentile score, on the USMLE Step 1 showed a statistically significant association with scores on the ABO-WQE, although the slope of the line of best fit (Pearson correlation coefficient) was only 0.45; however, the USMLE Step 1 did not show a statistically significant association with percentile scores on any of the OKAP examinations. There was a statistically significant association among all the OKAP examinations with the strongest between the second and third (Pearson correlation coefficient = 0.68) and the least strong between the first and third (Pearson correlation coefficient = 0.49). However, there was no association between percentile scores on any of the OKAP examinations and percentile scores on the ABO-WQE.
Table 1
Summary of variables
1999–2011
2004–2011
Variables
n
Mean
SD
Range
n
Mean
SD
Range
USMLE Step 1
31
64.6
30.7
8–96
20
78.9
20.1
23–96
Low pass (<85th)
19
9
High pass
12
11
OKAP 1
41
40.2
25.6
4–93
26
41.5
27.4
5–93
Fail (<30th)
17
11
Pass
24
15
OKAP 2
41
33
25.9
1–90
26
38.1
26.9
2–90
Fail (<30th)
23
12
Pass
18
14
OKAP 3
41
34.4
27
1–90
26
38.2
26.8
1–86
Fail (<30th)
22
16
Pass
19
10
ABO-WQE 1
41
55.2
25.6
8–96
26
63.8
20.2
29–96
Fail
9
0
Pass
32
26
Abbreviations: ABO-WQE 1, American Board of Ophthalmology Written Qualifying Examination, first attempt; OKAP, Ophthalmology Knowledge Assessment Program; SD, standard deviation; USMLE, United States Medical Licensing Examination.
Table 2
Pearson correlations among USMLE Step 1, OKAP, and ABO-WQE 1 percentile scores
Pearson
OKAP 1
OKAP 2
OKAP 3
ABO-WQE 1
USMLE 1
R
0.07
0.27
0.24
0.45
r
2
0.005
0.07
0.06
0.20
p
0.7
0.14
0.2
0.01
OKAP 1
R
1
0.66
0.49
0.35
r
2
0.43
0.24
0.12
p
< 0.001
0.001
0.33
OKAP 2
R
1
0.68
0.53
r
2
0.46
0.28
p
< 0.001
0.2
OKAP 3
R
1
0.49
r
2
0.24
p
0.23
Abbreviations: ABO-WQE 1, American Board of Ophthalmology Written Qualifying Examination, first attempt; OKAP, Ophthalmology Knowledge Assessment Program; USMLE, United States Medical Licensing Examination.
[Table 3 ] shows the odds ratio (OR) as well as the relative risk for both failing (RR−) and passing (RR+) the ABO-WQE in each of the following situations: scoring a low pass versus high pass (85th percentile or better, which correlates to a two-digit score of 99) on the USMLE Step 1; passing the OKAP examination in a given year of residency; passing every OKAP examination; passing two or more OKAP examinations; and passing at least one OKAP examination. Statistical significant association was greatest using a cutoff of 20th percentile on the OKAP examination among second-year residents and the ABO-WQE (p = 0.001), with an OR of 20.4, a RR− of 20.4, and an RR+ of 1.8. A cutoff of 30th percentile on the second-year OKAP was also statistically significant (p = 0.003); however, because every resident at our institution who achieved this cutoff (as well as every resident who passed all OKAP examinations) went on to pass the ABO-WQE on first attempt, statistical analysis was limited to calculating the RR+ which was 1.64. Cutoffs of 30th and 40th percentile on the third-year OKAP examination were also statistically significant (p = 0.016 and 0.036, respectively), yielding an OR of 10.3 and 8, an RR− of 6.9 and 5.7, and an RR+ of 1.49 and 1.41, respectively. The highest positive predictive value for failure (probability of failing the ABO-WQE on first attempt if not achieving a given percentile on a certain OKAP examination) occurred on the second-year OKAP examination at a cutoff of 20th percentile and was only 0.47.
Table 3
Univariate associations between USMLE Step 1 and OKAP examinations and ABO-WQE 1
Variable
OR
95% CI
RR (fail)
95% CI
RR (pass)
95% CI
p -value
USMLE 1 (high pass)
5.1
0.53–48
3.8
0.52–28
1.3
0.94–1.9
0.13
OKAP 1
1.2
0.26–5.2
1.1
0.35–3.6
1.0
0.74–1.4
0.84
OKAP 2
[a ]
[a ]
[a ]
[a ]
1.6
1.1–1.6
0.003
OKAP 3
10
1.2–92
6.9
1.0–50
1.5
1.1–2.1
0.02
Pass all OKAPs
[a ]
[a ]
[a ]
[a ]
1.4
1.1–1.8
0.03
Pass two or more OKAPs
10
1.2–92
6.9
1.0–50
1.5
1.1–2.1
0.02
Pass at least one OKAP
2.9
0.6–14
2.2
0.71–6.7
1.3
0.81–2.1
0.17
Abbreviations: ABO-WQE 1, American Board of Ophthalmology Written Qualifying Examination, first attempt; CI, confidence interval; OKAP, Ophthalmology Knowledge Assessment Program; OR, odds ratio; RR, relative risk; USMLE, United States Medical Licensing Examination.
a Unable to calculate.
[Table 4 ] compares the mean, standard deviation, and median percentile scores on the USMLE Step 1 and various OKAP examinations between residents who passed the ABO-WQE on first attempt and residents who failed on first attempt. The mean percentiles between residents who passed and those who failed were statistically significant in all examinations except for the OKAP examination in the first year of residency.
Table 4
Scores on various standardized tests classified by result on the ABO-WQE 1
ABO-WQE
Fail
Pass
p -value
Mean (SD)
Median (IQR)
n
Mean (SD)
Median (IQR)
n
USMLE 1
28.6 (28.5)
17 (14–29)
7
75 (22.6)
79.5 (74–92)
24
0.004
OKAP 1
32 (16.4)
30 (20–48)
9
42.5 (27.4)
41 (21–60)
32
0.16
OKAP 2
11.4 (8.8)
10 (15–18)
9
39.1 (25.9)
36.5 (19–60)
32
< 0.001
OKAP 3
17.2 (17.3)
13 (3.5–27)
9
39.3 (27.4)
35 (17–58)
32
0.008
Abbreviations: ABO-WQE 1, American Board of Ophthalmology Written Qualifying Examination, first attempt; IQR, interquartile range (25th–75th percentile); OKAP, Ophthalmology Knowledge Assessment Program; USMLE, United States Medical Licensing Examination.
[Fig. 1 ] shows the receiver operating characteristic curves for percentile scores on the USMLE Step 1, OKAP year 1–3 examinations, the minimum OKAP percentile, maximum OKAP percentile, and mean OKAP percentile score. The USMLE Step 1 showed the greatest area under the curve (0.896), and the second-year OKAP examination showed the greatest area under the curve (0.830) among the OKAP examinations, while the first-year OKAP examination showed the least (0.624).
Fig. 1 Receiver operating characteristic curve computed for predictive ability of United States Medical Licensing Exam (USMLE) Step 1 and Ophthalmology Knowledge Assessment Program (OKAP) compared with the American Board of Ophthalmology Written Question Exam.
Discussion
Medical residency training programs are entrusted with the education and development of future doctors and are expected to produce competent physicians. One measure of competence is national board certification in a specialty. The national American Board of Ophthalmology certification rate for graduates between 1999 and 2003 has been reported to be 78.8% within 2 years and 70.8% on the first try; and the first-try ABO-WQE pass rate was reported to be 80.3% in the same time period.[5 ] Johnson et al reported a first-time ABO-WQE pass rate of 72.6 and 75.9% within 3 years in their study of graduates from 1996 to 2006.[3 ] Our study, which reports a first-time ABO-WQE pass rate of 78 and 87.8% within 3 years, appears to be in accordance with these numbers.
The high national pass rate on first attempt of the ABO-WQE makes it difficult to identify indicators of failure that can be used to help identify residents at risk. Various analyses, including this one, have demonstrated a difference in performance on OKAP examinations between residents who pass the ABO-WQE on first attempt and those who do not, which would suggest that these conclusions can be applied to a population; however, our results suggest that these conclusions have limited application to an individual's performance. The study of Lee et al, which is the largest to date, found that the OKAP year 3 examination was the best indicator of ABO-WQE pass rate; however, according to their model, even those residents who score in approximately the 4th percentile still had a 50% pass rate, i.e., the positive predictive value for failing the ABO-WQE was 50%.[4 ] Likewise, in our model, we found the highest indicator of failing the ABO-WQE on first attempt was not achieving a score of 20th percentile on the second-year OKAP examination; however, the positive predictive value for failing the ABO-WQE was 47%, meaning 53% residents still went on to pass the ABO-WQE. These numbers seem significant, except that the positive predictive value for failing the ABO-WQE among graduating ophthalmology residents nationwide, independent of performance on OKAP examinations, is 19.7% on first attempt and approximately 30% overall.[5 ] This would suggest that the OKAP is not a strong model for projecting performance on the WQE, which is likely due to the low positive predictive value of failure. Based on the current set of data, the OKAP is not a useful tool for identifying residents at risk for not passing the ABO-WQE.
In addition, medical residency training programs have the arduous task of trying to select the applicants who are best suited for the field with the highest promise of success. There are few standardized evaluations available during the selection process to compare applicants; the USMLE Step 1 is one of these benchmarks. A multicenter analysis demonstrated a statistically significant difference in mean USMLE Step 1 scores among residents who passed the ABO-WQE on first attempt and those who failed.[4 ] When we performed this analysis in our cohort, there was also a statistical difference (p = 0.004) with the mean USMLE Step 1 percentile being 28.6th among residents who failed the ABO-WQE and 75th among those who passed ([Table 3 ]). Johnson et al reported in their study that performance on the USMLE Step 1 correlated to performance on the OKAP examinations in decreasing strength, but there was no correlation to performance on the ABO-WQE.[3 ] When we performed a similar analysis using Pearson correlations, we also did not find a correlation between USMLE Step 1 and ABO-WQE performance ([Table 2 ]). However, we could not demonstrate a correlation between performance on the USMLE Step 1 and OKAP examinations.
Our study was limited by a small sample size. A second limitation is the basic assumption that OKAP and ABO-WQE have the same goal. However, the purpose of the OKAP is to assess a resident's fund of knowledge as compared with peers to help identify weaknesses which the resident should improve upon, while the purpose of the ABO-WQE is to determine whether a physician has the necessary fund of knowledge to practice patient care in a safe manner at the community standard. While the goal of the examinations is not the same, thus making them not ideal for comparison, they both function as a standardized examination testing fund of knowledge judged against one's peers. Certain assumptions were also made for the ease of statistical analysis, such as the assumption that USMLE Step 1 scores obey a Gaussian distribution. Another assumption was the pass rate for the OKAP examination of 30th percentile; the OKAP examination has no cutoff for a passing score; however, to reduce the effect of this assumption, other cutoffs were also evaluated for possible significance. It would be interesting to see if any of the markers identified in the study were indicators of performance on the American Board of Ophthalmology oral examination; however, at this time that information is not available at our institution.
