Relationship between Didactic Attendance and Average Ophthalmic Knowledge Assessment Program Performance

Abstract

Background The Accreditation Council for Graduate Medical Education's ophthalmology-specific program requirements indicate that a resident must participate in 360 hours of educational activities over the course of residency. The requirements allow a variety of options by which this educational process may occur. Furthermore, the current generation of learners has used a large array of learning tools in their careers: digital media, online resources, question banks, flipped classrooms, problem-based learning, and asynchronous lectures in addition to the traditional face-to-face didactic lectures.

Objectives This article determines if the traditional face-to-face didactic lectures play a significant role in the education of ophthalmology residents, attempts to quantitate this role, and estimates the relative importance of this method.

Methods Lecture hours beginning immediately after an Ophthalmic Knowledge Assessment Program (OKAP) examination and up to the subsequent exam was tabulated for 40 residents over 8 years. From 2012 through 2019 each of the residents took the OKAP exam one to three times. The average overall OKAP score for each resident was then regressed against his or her logged hours of didactics that year. This rate of average OKAP score to lecture hours was then evaluated for significant differences on the basis of postgraduate year (PGY). “STATA 13.1 IC,” was used for all statistical analysis.

Conclusion Traditional synchronous didactic lectures play a statistically significant role in the education of ophthalmology residents. For each 3.5 hours of lectures per year there is a 1% increase in the average OKAP score, p = 0.0064. However, the amount of variability explained by this single factor is low, R ² = 0.067, and other factors are presumed to play a large part in learning. PGY subgroup analysis showed significant difference in the OKAP performance to lecture hour relationship, the number of lecture hours attended, and the OKAP performance.

Keywords

OKAP - Ophthalmic Knowledge Assessment Program - face-to-face lectures - traditional didactic lecture - resident education - synchronous learning - asynchronous learning - didactic lecture

With an increased number of learning platforms and continual changes in the way residents learn, the relative role of traditional lectures in conveying ophthalmic knowledge is of interest. Faculty members put many hours into preparing and updating lecture content, which becomes rapidly outdated. Therefore, optimization in producing learning content is of value.

Studies assessing the relative efficacy of problem-based learning versus lecture-based learning have shown varying results for short-term and longer-term learning.[1] [2] The contemporary learner has been exposed to question banks, preparatory courses for the United States Medical Licensing Exams, and a variety of tools from online resources to simulation laboratories. With an ever-increasing quantity of knowledge and skills required to practice ophthalmology and a fixed term of learning, it is worthwhile to determine what is most efficient in transferring medical knowledge and skills. In ophthalmology, this knowledge may be estimated with the Ophthalmic Knowledge Assessment Program (OKAP). It is a 260-item multiple-choice test that is administered to ophthalmology residents in each year of training. It is designed to measure the ophthalmic knowledge of residents, relative to their peers, to facilitate the ongoing assessment of resident progress and program effectiveness. Residents are assessed relative to their peers in 13 categories plus the overall average performance. This is expressed as a percentile rank relative to the training year peers.

In this study, we attempt to estimate the value per hour of traditional slide-based face-to-face lectures with respect to acquisition of knowledge using the OKAP's overall average percentile score as the proxy.

Purpose

To assess the efficacy of traditional slide-based face-to-face didactic lectures on ophthalmic knowledge acquisition as estimated through the average OKAP performance.

Methods

This study was reviewed by the Institutional Review Board (IRB) at the University of Arizona and IRB oversight was deemed unnecessary. We reviewed yearly records of didactic attendance from 2011 through 2019. This corresponded with 8 years of OKAP test results available for 12 residents per year with a total of 40 residents being represented. The sum of total hours, from immediately after an OKAP examination until immediately before the following exam, were recorded per resident along with their corresponding average percentage OKAP scores. All lectures during this period were of the traditional slide-based face-to-face lecture format.

A simple regression analysis was used to determine if a linear relationship existed between average OKAP scores and the total of yearly didactic hours per resident. This relationship was also subsequently subanalyzed by postgraduate year (PGY).

Finally, the effects of the categorical variable, PGY level, on the OKAP performance to lecture hour relationship were assessed using one-way analysis of variance (ANOVA) and pairwise comparisons using the Tukey's post hoc test. The ANOVA and Tukey's test were also performed to determine PGY associations with the number of didactic hours as well as the OKAP scores. STATA 13.1 IC (Stata Corp, College Station, TX) was used for all statistical analysis.

Results

Using a linear regression model of average OKAP scores and yearly total hours of didactics there was a statistically significant positive slope of 0.285%/hour of instruction per year (95% confidence interval [CI] 0.08–0.49) ([Table 1]). Restated, for every 3.5 hours (95% CI 2–13 hours) of didactics a 1% increase in the average OKAP score was found. The low level of correlation is demonstrated by the adjusted R ² of 0.067 ([Table 1]) and by the scatter plot ([Fig. 1]).

Fig. 1 Average Ophthalmic Knowledge Assessment Program (OKAP) score in percent versus yearly hours of didactic lectures for individual residents from 2012 to 2019. Slope = 0.285, p = 0.006, R ² = 0.067.

Table 1 Regression of average OKAP scores against yearly total of didactic hours per resident

All PGY[a]	SS	df	MS	Number of observation	96
Model	6,404	1	6,404	Prob > F	0.0064
Residual	77,537	94	825	R 2	0.076
Total	83,941	95	884	Adjusted R 2	0.067
OKAP	Coefficient	Std. Err.	t	p > | t|	[95% CI]
Hours	0.29	0.10	2.8	0.006	0.082, 0.49
Constant	9.1	15.2	0.6	0.55	–21, 39

Abbreviations: CI, confidence interval; df, degrees of freedom; MS, mean of sum of squares; OKAP, Ophthalmic Knowledge Assessment Program; PGY, postgraduate year; SS, sum of squares; Std. Err., standard error of the mean.

^a Represents PGY 2, 3, and 4 residents.

This relationship was driven predominantly by the PGY 4 group which was the only PGY group, on subgroup analysis, to show a statistically significant slope, 0.36%/hour of instruction per year, p = 0.0024 ([Table 2]). The PGY 4 group also showed the greatest adjusted R ² = 0.24 ([Table 2]).

Table 2 Regression of average OKAP scores versus hours of didactic instruction by PGY year

PGY 2	SS	df	MS	Number of observation	32
Model	2,611	1	2,611	Prob > F	0.0851
Residual	24,710	30	824	R 2	0.096
Total	27,320	31	881	Adjusted R 2	0.065
OKAP	Coefficient	Std. Error	t	p > |t|	[95% CI]
Hours	–0.51	0.29	–1.78	0.085	–1.1, 0.076
Constant	129	41	3.12	0.004	44, 213
PGY 3	SS	df	MS	Number of observation	32
Model	1,643	1	1,643	Prob > F	0.18
Residual	26,607	30	887	R 2	0.058
Total	28,250	31	911	Adjusted R 2	0.027
OKAP	Coefficient	Std. Error	t	p > |t|	[95% CI]
Hours	0.32	0.23	1.36	0.184	–0.16, 0.80
Constant	7.4	38	0.2	0.847	–70, 85
PGY 4	SS	df	MS	Number of observation	32
Model	5,394	1	5,394	Prob > F	0.0024
Residual	14,743	30	491	R 2	0.27
Total	20,138	31	650	Adjusted R 2	0.24
OKAP	Coefficient	Std. Error	t	p > |t|	[95% CI]
Hours	0.36	0.11	3.31	0.002	0.14, 0.58
Constant	–11	15	–0.7	0.49	–41, 20

Abbreviations: CI, confidence interval; df, degrees of freedom; MS, mean of sum of squares; OKAP, Ophthalmic Knowledge Assessment Program; PGY, postgraduate year; SS, sum of squares; Std. Error, standard error of the mean.

There was a relationship between PGY level and both yearly hours of didactics and OKAP scores. In the former, PGY 3 residents attended a statistically significant larger number of didactic hours than both PGY 2 and PGY 4 residents ([Table 3]). PGY 3–PGY 2 = 18 hours, p = 0.025, PGY 3–PGY 4 = 25 hours, p = 0.001, and PGY 4–PGY 2–6.6 hours, p = 0.97.

Table 3 ANOVA of yearly didactic hours by PGY level with pairwise comparison of PGY levels using Tukey's test

ANOVA hours vs. PGY, sequential
Number of observations = 96			R 2 = 0.13
Root mean square = 27			Adjusted R 2 = 0.11
	Seq SS	df	MS	F	Prob >  F
Model	10,372	2	5,186	7.03	0.0014
PGY	10,372	2	5,186	7.03	0.0014
Residual	68,574	93	737
Total	78,946	95	831
PGY	Margin	Std. Error	95% unadjusted CI
2	142	4.8	132, 151
3	160	4.8	150, 169
4	135	4.8	126, 145
PGY	Contrast	Std. Error	Tukey's test t, p > |t|		95% CI unadjusted
3 vs. 2	18	6.8	2.65	0.025	1.9, 34
4 vs. 2	–6.6	6.8	–0.97	0.600	–23, 9.6
4 vs. 3	–25	6.8	–3.6	0.001	–41, –8.4

Abbreviations: ANOVA, analysis of variance; CI, confidence interval; df, degrees of freedom; F, Fischer's statistic; MS, mean of sum of squares; PGY, postgraduate year; Seq SS, sequential sum of squares; Std. Error, standard error of the mean.

Similarly, there was a statistically significant greater performance on the OKAPs by PGY 3 and PGY 2 relative to PGY 4 residents (PGY 3–PGY 4 21%, p = 0.012, PGY 2–PGY 4 18%, p = 0.034). PGY 2 and 3 were not significantly different, PGY 3–PGY 2 3%, p = 0.92 ([Table 4]).

Table 4 ANOVA of OKAP by PGY level with pairwise comparison of PGY levels using Tukey's test

ANOVA OKAP vs. PGY, sequential
Number of observations = 96			R 2 = 0.098
Root mean square = 29			Adjusted R 2 = 0.079
	Seq SS	df	MS	F	Prob >  F
Model	8,233	2	4,117	5.06	0.0082
PGY	8,233	2	4,117	5.06	0.0082
Residual	75,708	93	814
Total	83,941	95	884
PGY	Margin	Std. Error	95% unadjusted CI
2	56	5	46, 66
3	58	5	48, 68
4	38	5	28, 48
PGY	Contrast	Std. Error	Tukey's test t, p > |t|		95% CI unadjusted
3 vs. 2	2.75	7	0.39	0.92	–14, 20
4 vs. 2	–18.1	7	–2.54	0.034	–35, –1
4 vs. 3	–20.9	7	–2.93	0.012	–38, –4

Abbreviations: ANOVA, analysis of variance; CI, confidence interval; df, degrees of freedom; MS, mean of sum of squares; OKAP, Ophthalmic Knowledge Assessment Program; PGY, postgraduate year; Seq SS, sequential sum of squares; Std. Error, standard error of the mean; t, Student's t statistic.

Discussion

There are several cross-sectional studies in the literature that show a statistically significant positive correlation between the relative quantity of didactic lectures with medical learning.[3] [4] [5] These studies evaluated graduate medical learning in variety of disciplines and topics.

The study presented here provides additional evidence to suggest that traditional face-to-face lecture-based didactic sessions have a role in improving medical knowledge. In this case, there was a statistically significant relationship with the OKAP examinations but the correlation, as seen with other studies, was low suggesting that there are other factors playing a significant role in medical education.[3] The PGY 2 and PGY 3 subgroups showed no statistically significant correlation between the number of didactic lecture hours and OKAP performance. For PGY 4 residents, there was a statistically significant correlation suggesting that for every 2.8 hours (95% CI 1.7–7.3 hours) of lecture per year the average OKAP score increased 1%, p = 0.0024. This single factor explained approximately 24% of the variability for this group's average OKAP performance. For all three PGY levels taken together the R ² suggested that only 6.7% of the variability was explained by lecture hours and the slope for all three taken together showed that a greater number of hours were required, 3.5 (95% CI 2–13 hours), to increase the average OKAP performance by 1%. It is possible that repetition and a greater platform of ophthalmic knowledge in the senior year make lectures higher yield.

In the present study, there was a statistically significant association between PGY level and both yearly hours of didactics and OKAP scores. Here, the PGY 3 level showed the greatest number of didactic hours and the highest OKAP performance. There are several patterns in the program, from which this study was derived, to explain these PGY differences. The PGY 3 residents are sent to a multiday out of state review course approximately 1 week preceding the OKAP exam which increases the amount of dedicated and optimally timed learning relative to the other two groups. A new environment for learning with the away course may play a role as well. Also, residents interested in fellowships are often asked for their OKAP scores thus increasing the incentive to do well on the PGY 3 year test. Additionally, the PGY 2 residents arrive on July 1 undergoing on-boarding during the first few weeks and effectively obtain 8 months of didactics as compared with 12 months for the other two groups. Finally, it has been our experience that the senior residents do not value a high score on the OKAPs as greatly and find the demands of improving cataract proficiency and job interviews as competing factors to at-home study. This may also partially explain why mandatory lectures, thus responsible for a larger percentage of their studies, play a more significant role in the senior year.

The literature would suggest that complementary or alternative methods to traditional lecture-based presentations might be effective at improving learning as well as improving the environment for both teacher and student. Several alternative methods have been shown to be effective. One example, case-based small discussion groups, in connection with traditional lectures significantly improved test scores at the conclusion of two of the five topics investigated. This crossover study involved 170 medical students.[6] Similarly, computer-based learning was found to supplement face-to-face lectures in a study of 26 medical students learning dermato-oncology. Here, the test results from questions corresponding the computer-based portion of the curriculum were superior to those corresponding to the traditional lectures, p < 0.05.[7]

Furthermore, an online text-based learning system was shown to be significantly more effective, p = 0.012, when an aural accompaniment was present and also lead to a greater number of students seeking out additional study material.[8] Adding a problem-based component to the didactic curriculum has also proven effective. In a study with 95 nursing students randomly assigned to traditional lectures, lectures plus problem-based learning and solely problem-based learning, those with a component of problem-based learning did significantly better, p = 0.001, on testing immediately after and after weeks of delay.[9] Another creative adjunct to standard anatomy lectures involved the addition of live narrated interactive video streaming of live surgical procedures. In this nonrandomized study, 138 medical students performed significantly better on clinically related multiple-choice questions, p < 0.05, relative to fellow dental students who participated in traditional lectures alone.[10] It is also important when assessing the effect of a teaching intervention to consider the proximity of the intervention relative to the time of testing.[11] Finally, participant preference should be considered. For example, 44 ophthalmology residents were randomized into two groups. One group took the flipped glaucoma classroom and lecture-based ocular trauma classroom, while the other group took the flipped ocular trauma classroom and lecture-based glaucoma classroom. There was no difference in the posttest examination between groups and the residents and attendings rated the flipped classroom as more desirable. Eighty percent of the group suggested one flipped classroom per week as the optimal exposure to the flipped classroom venue.[2]

For additional consideration, there are advantages to synchronous and asynchronous learning and both lend themselves to traditional lectures. The Accreditation Council for Graduate Medical Education review committee for ophthalmology allows a great range in learning option to satisfy the 360-hour requirement over the course of a residency.[12] Emergency medicine has made use of asynchronous learning, as it is difficult to have everyone present at the same location at the same time. It allows on-demand viewing at one's own pace, stopping, starting, and reviewing as desired. It can be as simple as recording a lecture for later use, an adjunct to a flipped classroom, a prelude to problem-based learning, or it can allow review of periodic regulatory material that would otherwise fill valuable learning time. In a prospective randomized study comparing emergency medicine residents receiving synchronous versus asynchronous instruction, the synchronous group scored better initially but at the 10-week retest there was no difference in performance. The two formats had similar degrees of acceptability.[13]

The largest shortcomings of the study presented here were lack of randomization and the small sample size. Without randomization lecture hour attendance cannot be separated from potentially confounding variables such as interest in the subject, competing events, and other factors that might link lecture attendance with OKAP performance. Additionally, the small sample size was accentuated when subdividing further in the PGY group analysis. The sample size precluded looking at important factors such as lecturer, timing relative to OKAPS, Basic and Clinical Science Course section, innate individual test-taking skills, and call and rotation duties prior to the examinations. Finally, with the small size and single program makeup of this study, generalization to all ophthalmology programs or other specialties warrants caution.

Other factors worth investigating, that may prove useful in predicting the attainment of ophthalmic knowledge, might include determination of baseline ophthalmic understanding and test-taking skills and tracking what tools are used to study and the duration each is used. The many inputs from the learning tools combined with common assessment tools such as OKAPs, written board scores, and oral board pass rates complement artificial intelligence platforms such as the Kalman filtering process. With standardization of assessing surgical skill performance or any of the milestone it is possible, by pooling data from many programs, to project resident performances, detect problem areas sooner, and individualize learning plans. For example, pooled interprogram data might suggest that 30 hours (95% CI 20–35) of simulation training is required for capsulorhexis proficiency. This would be useful information before a resident embarks on a rotation as the primary surgeon in a high volume general ophthalmology rotation. This pooling of data has been successfully done with assessment of glaucoma progression[14] [15] as well as many unrelated disciplines with the commonality of multiple inputs, measurable outcomes, and many data points. Finally, providing the residents with statistically based feedback from learning studies can assist in changing behavior.

Conflict of Interest

None declared.

• References

• 1 Faisal R, Rehman KU, Bahadur S, Shinwari L. Problem-based learning in comparison with lecture-based learning among medical students. J Pak Med Assoc 2016; 66 (06) 650-653
• 2 Lin Y, Zhu Y, Chen C. et al. Facing the challenges in ophthalmology clerkship teaching: is flipped classroom the answer?. PLoS One 2017; 12 (04) e0174829
• 3 Wongjarupong N, Niyomnaitham D, Vilaisaktipakorn P, Suksiriworaboot T, Qureshi SP, Bongsebandhu-Phubhakdi S. Interactive laboratory classes enhance neurophysiological knowledge in Thai medical students. Adv Physiol Educ 2018; 42 (01) 140-145
• 4 Millis RM, Dyson S, Cannon D. Association of classroom participation and examination performance in a first-year medical school course. Adv Physiol Educ 2009; 33 (03) 139-143
• 5 Al Khaja KAJ, Tayem Y, James H, Jaradat A, Sequeira RP. Pharmacology and therapeutics resource session attendance and academic performance of pre-clerkship medical students in problem-based learning curricula. BMC Med Educ 2019; 19 (01) 269
• 6 Gali S, Shetty V, Murthy NS, Marimuthu P. Bridging the gap in 1(st) year dental material curriculum: a 3 year randomized cross over trial. J Indian Prosthodont Soc 2015; 15 (03) 244-249
• 7 Smolle J, Staber R, Neges H, Reibnegger G, Kerl H. Computer-based training in dermatooncology--a preliminary report comparing electronic learning programs with face-to-face teaching [in German]. J Dtsch Dermatol Ges 2005; 3 (11) 883-888
• 8 Ridgway PF, Sheikh A, Sweeney KJ. et al. Surgical e-learning: validation of multimedia web-based lectures. Med Educ 2007; 41 (02) 168-172
• 9 Salari M, Roozbehi A, Zarifi A, Tarmizi RA. Pure PBL, hybrid PBL and lecturing: which one is more effective in developing cognitive skills of undergraduate students in pediatric nursing course?. BMC Med Educ 2018; 18 (01) 195
• 10 Shiozawa T, Butz B, Herlan S, Kramer A, Hirt B. Interactive anatomical and surgical live stream lectures improve students' academic performance in applied clinical anatomy. Anat Sci Educ 2017; 10 (01) 46-52
• 11 Miller CJ, Aiken SA, Metz MJ. Perceptions of D.M.D. student readiness for basic science courses in the United States: can online review modules help?. Eur J Dent Educ 2015; 19 (01) 1-7
• 12 Review Committee for Ophthalmology, Frequently Asked Questions: Ophthalmology. Available at: https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/240_Ophthalmology_2020.pdf?ver=2019-02-19-121341-650. Accessed April 8, 2020
• 13 Jordan J, Jalali A, Clarke S, Dyne P, Spector T, Coates W. Asynchronous vs didactic education: it's too early to throw in the towel on tradition. BMC Med Educ 2013; 13: 105
• 14 Schell GJ, Lavieri MS, Helm JE. et al. Using filtered forecasting techniques to determine personalized monitoring schedules for patients with open-angle glaucoma. Ophthalmology 2014; 121 (08) 1539-1546
• 15 Kazemian P, Lavieri MS, Van Oyen MP, Andrews C, Stein JD. Personalized prediction of glaucoma progression under different target intraocular pressure levels using filtered forecasting methods. Ophthalmology 2018; 125 (04) 569-577

Publication History

Received: 30 July 2020

Accepted: 03 September 2020

Article published online:
26 November 2020