Keywords
electronic health record - EHR - research - resident - EHR training - primary care - learner - assessment
Background and Significance
Background and Significance
The Electronic Health Record (EHR) has improved many aspects of patient care, including quality, efficiency, and cost but has significantly impacted provider–patient dynamics and clinical workflow.[1]
[2] In a recent national survey, Graduate Medical Educators agreed that the EHR impacts the education of residents and fellows in several ways, especially in the importance of “balancing focus on electronic documentation with patient engagement,” and the need to teach EHR use through hands-on practice.[3] Many EHR training programs are technical, brief, and not customized to the learner's clinical specialty. Trainer experience and learner engagement vary. In our experience, residents do much of their EHR learning during clinical care time, adopting the habits and workarounds of their colleagues. As resident physicians are central to academic institutions, suboptimal EHR training can result in inefficiency and user frustration and the need for significant ongoing support.[4]
[5]
[6] Such delays interrupt patient care, increase workload, and residents have reported decreased available education time.[4] This is worrisome, as command of information technology is “vital in the modern practice of medicine.”[7]
Effective EHR learning programs incorporate social cognitive theory and adult learning principles.[8]
[9] Several studies suggest that enhanced EHR training positively impacts EHR adoption, efficiency, and patient care.[5]
[10] Enhanced EHR training has been investigated for large groups of residents across multiple specialties[4]
[11]
[12] and medical students,[13]
[14] but objective measurement of these interventions is absent or in early phases of exploration.[11]
[15]
[16] Most published literature on this topic only utilizes self-assessment methods.
Objectives
-
To design and implement an enhanced EHR learning curriculum that incorporates adult learning theory and multimodal educational tools relevant to the PGY1 resident.
-
To create and implement assessment tools for EHR proficiency and self-reported efficiency and self-efficacy.
Methods
This study was completed at a large academic pediatric practice. All new PGY1 residents received basic training taught by a nonclinical software specialist. Basic training occurred in a classroom setting with individual desktop computers and a large instructional monitor. Content was not specialty specific and was presented in a didactic fashion with occasional individual learner task practice. In the months following initial go-live in January 2012, EHR superusers conducted an informal needs assessment in the resident clinic. Superusers were attending physicians who had received additional EHR training, led site implementation and clinical support, and were regarded as local experts. Superusers noted that in the months after EHR adoption, preceptors were devoting approximately 30 to 60 minutes per clinic session to instructing residents in EHR use and correcting EHR errors, instead of providing clinical education. Superusers also noted widespread deficiencies in residents' knowledge of basic EHR tasks and decreased time spent on clinical care. In response to the needs assessment, superusers created an enhanced EHR training curriculum for incoming residents built upon social cognitive theory and adult learning principles frequently used in graduate medical education (see Appendix 1).
Our study was a prospective, single center cohort study of Pediatric PGY1s' EHR skills and attitudes before and after the enhanced EHR training curriculum. All new PGY1s were required to participate in the basic and enhanced training, and all were invited to participate in this study from the start. Written consent was obtained for those who agreed. The EHR utilized in our practice was NextGen Ambulatory EHR (QSI Management, LLC; Horsham, Pennsylvania, United States).
Enhanced Training Intervention
We designed an enhanced, three-phase EHR training curriculum to follow basic training (Appendix 2).
Phase 1 was designed to orient PGY1s to EHR layout and basic visit workflows. An EHR physician superuser led small group classes over 4 hours, reviewing a typical pediatric visit using demonstrations, learner practice activities and role plays, and entry of medications and vaccines. For example, a volunteer pretended to be a parent, using a script to give the history for a sick child, while PGY1 learners documented within the Histories template. In another example, PGY1s were provided a list of 10 frequently prescribed pediatric medications (such as amoxicillin suspension) and practiced creating prescriptions within the medication module. All EHR terminology was defined in language familiar to learners.
Phase 2 was held within the PGY1's continuity clinic to situate an EHR practice experience within the actual clinical setting. Residents were provided 60 minutes during the first week's orientation to strengthen and reinforce EHR skills through role playing with data entry. For example, a pair of PGY1s would use an exam room to role play as patient and physician, with the physician entering data into the Histories, Review of Systems, Physical Exam, and Assessments templates. PGY1s also utilized this time to practice creating prescriptions that were then reviewed by an attending physician, and later set up saved “favorite” medications, My-Phrases, and templates within the system.
Phase 3 was held during the PGY1s' second clinic week, focusing on skill application with supervision and feedback. As real-time, one-on-one EHR training in the work environment has been shown to enhance performance,[6] PGY1s observed two patient care visits performed by their senior residents (PGY3) and then acted as scribes for subsequent visits. As PGY3 residents had been using the system for 2 years, they served as experienced peer educators to support new learners' skill acquisition during this session.[8] PGY1 residents also entered orders and completed electronic forms under supervision during the PGY3 precepting time.
Assessment of Residents
EHR skills were measured through a proficiency assessment tool, and surveys measured self-reported EHR efficiency and self-efficacy. The proficiency assessment and surveys were administered together on two separate occasions: following basic training (pretest) and following enhanced training (posttest, [Fig. 1]).
Fig. 1 Schematic of curriculum and evaluation.
Proficiency Assessment Tool
To our knowledge, there exists no validated, widely used measure of EHR proficiency. To create our own measure and establish that its content represented the appropriate skills,[17] six typical EHR tasks for pediatric primary care were assigned to experienced clinicians for evaluation ( Appendix 3) and adjusted so that they could be completed within 6 minutes. Each task was given a score of 0 (not done), 1 (partially done), or 2 (completely done) and equally weighted toward a total score from 0 to 12. Each task was preassigned a detailed definition of the three possible scores, including attention to quality of content. At the pre- and posttests, each PGY1 was provided 6 minutes to complete the six tasks. Results were scored by a clinician blinded to curriculum content and study objectives. Mean group scores were utilized for analysis.
Efficiency and Self-Efficacy Surveys
Since no single, validated, widely used tool existed to measure these data, question themes were culled from the literature. Questions were piloted with physicians in the practice (details in Appendix 3). Five efficiency questions used the stem, “I can correctly complete [EHR task] without asking someone for help...” with a 4-option Likert response scale. Five self-efficacy questions used the stem, “How confident do you feel doing the following tasks...?” with a 4-option Likert response scale.
Analysis
For pre- and postcurriculum results, paired t-tests were employed to compare mean proficiency assessment scores, and the Wilcoxon Signed Rank Test (WSRT) compared the survey differences. Only assessments with matching pre- and postresponses were included. Survey data were imported into SPSS for analysis. Assuming 25 participants, we estimated 80% power to detect a change of ≥ 1.5 points on each scale with an estimated standard deviation of 2.5 and an α of 0.05 using a 2-sided one sample t-test.
Results
The incoming class of 26 PGY1s initiated the basic and enhanced training and consented to study participation. One PGY1 could not complete the training due to illness, and six individuals did not complete both pre- and posttests. This resulted in a sample size of 19 of 26 or 73%.
Proficiency Assessment
The mean total proficiency scores improved with statistical significance from pre- to posttesting (6.8–7.8, p = 0.048; [Fig. 2]). Of the six items, there was statistically significant improvement in medication entry (mean: 1.0–1.7, p = 0.02) and completing school forms (mean: 0.2–0.7, p = 0.03) with nonsignificant changes in the remaining items (adding vital signs, adding allergies, adding diagnoses, and ordering vaccinations).
Fig. 2 Mean proficiency assessment scores pre- and postcurriculum for six tasks and total for all tasks. *p < 0.05.
Efficiency and Self-Efficacy
Residents reported a low sense of efficiency and self-efficacy on the pretest. For pretest efficiency, only a minority of residents reported that they could correctly complete tasks “always” or “most of the time” ([Fig. 3]). For pretest self-efficacy, a minority of residents felt “very confident” or “somewhat confident” in three of the tasks, although they reported confidence while adding allergies or documenting a telephone call ([Fig. 4]). In the posttest, most residents reported a higher sense of efficiency than on the pretest, including vaccine ordering (WSRT, p < 0.05), complete blood count (CBC ordering; WSRT, p < 0.05), school physical form completion (WSRT, p < 0.05), and updating patient location (WSRT, p < 0.05; [Fig. 3]). History taking did not improve significantly (WSRT, p = 0.074). Residents reported more self-efficacy on the posttest for adding saved phrases (WSRT, p < 0.05), but the remainder reported less or stayed the same, including documenting a telephone call (WSRT, p < 0.05), adding allergies (WSRT, p = 1.00), ordering a rapid strep test (WSRT, p = 0.49), and locating test results (WSRT, p = 0.22; [Fig. 4]).
Fig. 3 Efficiency survey results pre- and post-curriculum for five tasks. Question stems, “I can correctly complete (task) without asking someone for help…” *p < 0.05.
Fig. 4 Self-efficacy survey results pre- and postcurriculum for five tasks. Question stems, “How confident do you feel doing the following tasks?” *p < 0.05.
There were similarities between results for those activities that were measured by both the surveys and the proficiency test. For updating allergies, most respondents reported confidence and had high proficiency scores at both the pre- and postassessments ([Figs. 2] and [4]). For ordering vaccines, respondents reported very low efficiency at the pretest and made modest improvements on the posttest ([Fig. 3]). In a similar pattern, proficiency scores were low and did not improve ([Fig. 2]). For school form completion, residents first expressed low efficiency and had low proficiency scores, and both measures significantly improved from pre- to postassessments ([Figs. 2] and [3]).
Discussion
As suspected, participating PGY1 residents did not report substantial efficiency or self-efficacy after basic EHR training. To develop an enhanced curriculum for the use after basic training, we incorporated additional adult learning theory, simulation, physician trainers, and one-on-one experiences.
These results suggest that a three-phased enhanced training curriculum may have improved resident EHR efficiency and some measures of proficiency. These findings are relevant given that EHR competence is now a basic clinical skill, and residents who lack proficiency and efficiency may struggle to perform clinical tasks. Although medicine and medical education are highly regulated fields, the literature about measuring EHR skill acquisition is scarce. This article adds to the field by describing a novel way to assess EHR skill acquisition. This study could contribute to future development of standardized training and objective assessment of EHR provider–users across multiple primary care settings. Of note, the biggest resource necessary for our intervention was physician time (4 × 2 = 8 hours), as EHR superuser attending physicians delivered the curriculum. This time resource, however, was less than the estimated amount of time required for on-the-spot EHR assistance provided by attending physicians for residents prior to this study.
Certain outcome measures showed no improvement or worsened. Although two individual proficiency scores and the average proficiency scores showed improvement, several scores worsened, which may reflect differing complexity of particular tasks. Additionally, the three of the four tasks that showed no improvement in proficiency were relatively simple tasks (vital signs, allergies, and diagnosis) that had high initial scores. It may be that learners achieved mastery after basic training and therefore had little room to improve further. Worsening self-efficacy may be due to the complex user interface of the EHR application. Also, as large parts of phases 2 and 3 were peer-to-peer, the quality of the training interaction was not established. In addition, it is possible that experienced users in phase 3 passed on maladaptive workarounds, worsening learner scores. Lastly, residents also may reach a “saturation point” of new information during a short but demanding orientation period.
Related studies published on this topic have mixed results. Dastagir and colleagues surveyed clinicians already experienced in their EHR who received a proficiency training course. Comparing pre- and posttraining surveys, they demonstrated improvements in self-assessed efficiency.[5] Vuk and colleagues demonstrated improved self-efficacy assessed before and after simulation training,[10] which differed from our self-efficacy results. Jalota and colleagues conducted a real-time peer training intervention with physicians already experienced in their EHR and showed improved documentation efficiency over the control group.[6] Reis and colleagues demonstrated no improvement in attitude, sense of competence, or observed proficiency when comparing a simulation based training to traditional lecture format.[12]
Our study had several limitations. Self-report of efficiency might not have reflected actual efficiency. Our sample size was small, and performance at a single center might have limited its external validity. A lack of a control group limited our ability to attribute changes to the intervention instead of increased training exposure alone (maturity threat). A selection bias might have been introduced by including only new residents, who typically were younger and more comfortable with technology than other physicians. The majority of our assessments dealt with data entry tasks, even though data extraction and interpretation were important EHR skills. Although our curriculum was adapted to our EHR application, it could be easily adapted to other systems.
Single group pre-and posttest models, such as ours, are at a risk of testing threat, in which exposure to the pretest influences participants' learning, thereby exaggerating their performance on the posttest. Our proficiency assessment and surveys evaluated slightly different sets of EHR tasks, therefore evaluating the exact set of tasks through the lens of both the surveys and the proficiency tool might have allowed the curriculum to be refined and focused on tasks requiring more instruction to achieve mastery. The assessment tools also deserve more thorough validation, including against clinical outcomes. However, this study was an attempt to integrate adult learning principles into EHR training and describe a method of EHR skill assessment in a manner that could be replicated. Study leaders received verbal feedback from colleagues describing stronger PGY1 EHR skills and the additional time available for precepting and clinical education after this intervention. Future studies could formally measure this preceptor feedback and measure time available for EHR teaching.
Next steps in this project would involve pretraining baseline assessments and further development of the proficiency measure. Recruiting a control group at a comparable institution, repeating the posttest at a later date, and monitoring improvements in accuracy and efficiency during subsequent clinical care would enrich our understanding of the effects of the intervention.
Conclusion
This study details an EHR training intervention that may improve residents' EHR skills; it proposes several measures of EHR skill proficiency and efficiency that may be studied further. Ultimately, an enhanced EHR training curriculum should strengthen residents' EHR skills and decrease interruptions, allowing residents and preceptors to prioritize direct patient care and build clinical knowledge.
Clinical Relevance Statement
Clinical Relevance Statement
Residents and EHRs build the backbone of academic medical institutions; however, resident EHR training is often overlooked. This study presents a novel, enhanced EHR training intervention designed with adult learning principles, and measures learners' proficiency, efficiency, and self-efficacy. Significant improvements in efficiency and some measure of proficiency are noted, which could support streamlined workflows, patient care quality, and time for clinical education.
Multiple Choice Questions
Multiple Choice Questions
-
Training curricula for clinicians to use an EHR should include
-
A fast-paced style to keep the learner's attention
-
Careful attention to the concerns about the application raised by the learners
-
Well-trained mock patients
-
Active learning techniques using real-life scenarios
-
When compared with standard EHR training, enhanced EHR training
-
Can positively impact the efficiency of clinicians' EHR use
-
Incorporates adult learning principles and/or social cognitive theory
-
Can require more time to complete
-
All of the above
Answers
-
Correct answer is D, Active learning techniques using real-life scenarios. Adult learning theory uses real-life scenarios and active engagement of the learner. The quality of the mock patient, being fast paced to keep the learner's attention and time spent on learners' concerns of the EHR application design are less important components of EHR training for medical providers.
-
Correct answer is D, All of the above. Incorporation of adult learning principles can result in a longer training period when compared with out-of-the-box training. However, evidence suggests that this could result in increased EHR user efficiency.
Description of Need Assessment and Curriculum Creation
As an informal needs assessment, EHR superusers spoke with a majority of attending preceptors inquiring about residents' proficiency or lack of proficiency with the various EHR workflows, types of errors made by residents in the EHR system, and what fraction of attending physicians' time was spent teaching the EHR instead of teaching clinical skills. Preceptors responded that residents were unfamiliar with the documentation of procedures, had difficulty remembering how to e-prescribe medications, and were not familiar with ordering point-of-care tests or utilizing the form generator, among other challenges. Additionally, 2 months after the initial go live, superusers, who also served as regular clinic preceptors, noted approximately 30 to 60 minutes per half-day clinic session devoted by preceptors to EHR instruction instead of clinical teaching about routine pediatric care, disease, or clinical skills. Superusers received similar, although informal, comments from colleagues about this matter. Superusers also spoke with one or two residents during each “clinic day” (Monday–Friday) regarding components of the EHR in which they felt that they needed additional training. Residents responded that they felt a requirement for additional training on how to order point-of-care and send out laboratory tests, document telephone calls, e-prescribe complex medications, capture history in real time, and generate commonly used forms. Additionally, residents reported having to physically go and find a staff person or colleague if they had a question about the EHR, which was time consuming.
Superusers reviewed an original list of all existing workflows in the current clinical practice and ordered them chronologically in relation to when they typically occur during patient care. Each workflow was then broken down into discrete steps, and the steps were written into the verbal curriculum script using plain language, forming the backbone of the enhanced curriculum. Particular attention was paid to the tasks in which current residents had reported inadequate training.
Next, superusers reviewed tools held as best practices in adult medical education, namely, the use of standardized patients or role playing to practice a new skill, real-time skill quizzing with instructor feedback, opportunities for self-directed practice, learning sessions conducted within actual clinical spaces, and the opportunity to shadow experienced peers performing complex tasks[9]. In addition, we reviewed published work on the role of social cognitive theory within EHR training,[10] which suggested that successful EHR training programs emphasized the positive potential of EHRs upon patient outcomes, incorporated active learning activities, identified positive role models for EHR use in the clinical environment, and incorporated an understanding of the characteristics of the community within which learners may eventually use the EHR.
In a deliberate, step-wise fashion, adult learning tools were carefully integrated into the list of clinical workflows, with careful attention paid to task complexity and the circumstances of real-life practice. For example, a parent/doctor role play script was written to be performed as PGY1s included learning the Histories Template, with instructions for learners to watch the actors and document the history of present illness (HPI). A practice set of 10 prescriptions was written to be used for independent practice prior to instructor feedback. Superusers created rhymes and humorous phrases as memory aids for tasks with multiple steps. The curriculum was then divided into phases (see Appendix 2) with opportunities for advanced practice of key workflows provided in a three-phase curriculum setup. Finally, introductory material was created to welcome learners, place the EHR within the context of patient care and their graduate education, and identify individuals in the clinical setting who could provide support during their EHR use.
Detailed Description of Enhanced Training, Phases 1, 2, and 3
Phase 1 of enhanced training was designed to orient PGY1s to the EHR layout and the basic workflows of ambulatory pediatric visits. Two classes (14 residents each) ran concurrently in identical training classrooms during the second week of residency orientation. Phase 1 instruction was led by general pediatrics and EHR physician superusers, who had advanced EHR experience and training. Both instructors trained from identical scripts to minimize variations. PGY1s were instructed to ask the instructors questions at any time and notify the teacher in the event of confusion or difficulty.
The phase 1 training followed the typical sequence of events of a well-child visit. The “visit” was divided into 15 sections (reason for visit/chief complaint, vital signs module, social history, physical exam, assessments, ordering laboratories, plan details, referrals, immunization modules, anticipatory guidance, procedure module, confidential teen history, medication module, document library, and laboratory results). For each section, the teacher would introduce the section of the visit and its purpose, demonstrate how to navigate to the appropriate area in the EHR, and demonstrate three times how to document in or use the functions of that section. The teacher would then instruct students to navigate to the section and have them perform a particular documentation or function within that section. Other smaller sections were covered if time allowed.
Additional exercises were given to reinforce skills in certain sections. For example, in the Social History section, volunteers used scripts to role play as parents while being interviewed about the family's home circumstances, and students typed the details into the Social History section. For the medication module, PGY1s were provided five pediatric medication examples to enter into the module. For the confidential teen history, volunteers role played as teens while being interviewed about their risk-taking behaviors, and PGY1s typed their data into the teen section. PGY1s were also provided a short list of immunizations to order over a 5-minute period. Additional teaching tools included props, humor, and short rhymes to help with memory.
Phase 2 occurred 1 to 4 weeks after phase 1 and was designed to situate an EHR practice experience within the actual clinical setting. This phase took place within the ambulatory primary care clinical space where PGY1s have their weekly continuity clinic. As part of the traditional residency orientation, PGY1s spent two dedicated afternoons in this space to meet the staff and had a tour, and phase 2 occurred on the first of these days. PGY1s were provided a workbook detailing their assignments. For the first assignment (40 minutes), residents formed groups of two and were provided two scripts of typical pediatric patient histories. They were instructed to simulate a typical clinical encounter by using an empty exam room and logging into the EHR system, where a “dummy” chart had been created for training purposes. With one resident reading from the parent script, the other resident would role play as the physician, and ascertain the patient's medical history while entering it into the EHR system. They would then repeat this exercise using the second script and reversing the roles. Attendings and upper-year residents familiar with the system were available in the event of questions or problems during the exercise.
For their second assignment (10 minutes), residents were provided a list of seven common medications with specific dosages and instructed to create the prescriptions within the EHR, print them, and have the prescriptions reviewed by an attending physician for accuracy and feedback. Their final assignment (10 minutes) was to set up and save their preferred “favorite” settings in the system, for example, designating frequently used medications in an easy-to-find folder.
Phase 3 of enhanced training allowed PGY1s to apply their EHR skills with close supervision and feedback. This phase occurred 1 to 4 weeks after phase 2. For the first 2 hours, PGY1s shadowed an upper-year resident during two routine visits. During the third visit, PGY1s scribed the history into the EHR in real time while the upper-year resident conducted the visit. During the precepting session with the attending physician, the PGY1 was closely supervised by the attending physician while updating the in-room patient tracking status and inputting necessary orders for the patient (for example, vaccines, laboratory tests, and referrals). The resident was also instructed to populate and generate the patient's school physical forms in the EHR. Immediate feedback was provided to the PGY1 by the attending physician on form accuracy and order correctness. Residents used any remaining time to set up custom “favorites” within the system and practice making common patient health documents.
Proficiency Assessment and Surveys
Proficiency Assessment Tasks*:
(*Pretest tasks. Posttest tasks were similar but used different pulse value, allergy, diagnosis, etc.)
-
Your patient's pulse is 111 beats/minute. Add this heart rate to the vital signs template and save it.
-
Your patient mentions a peanut allergy. Add this allergy to the allergy module. Indicate that the allergy is severe.
-
You diagnose your patient with ASTHMA. Add this to today's assessments.
-
In the medication module, prescribe albuterol HFA, 2 puffs q4h PRN cough and print the prescription.
-
Your patient requests a child health assessment. Create a child health assessment. Indicate the asthma history and the peanut allergy in the appropriate sections. Also, in the appropriate section, type that she/he needs to have albuterol and an EpiPen readily available.
-
Your patient needs a hepatitis A vaccine. Order the HepA vaccine and task to the appropriate recipient.
Survey
The following questions ask about your experience completing tasks in the NextGen EHR system. Choose the answer that best represents your experience at this point in your training.
-
Q1. I can correctly order a vaccine without asking someone for help.
-
○ ALWAYS (1)
-
○ MOST OF THE TIME (2)
-
○ SOME OF THE TIME (3)
-
○ ALMOST NEVER (4)
-
Q2. I can correctly order a CBC without asking someone for help.
-
○ ALWAYS (1)
-
○ MOST OF THE TIME (2)
-
○ SOME OF THE TIME (3)
-
○ ALMOST NEVER (4)
-
Q3. I can correctly create a child health assessment without asking someone for help.
-
○ ALWAYS (1)
-
○ MOST OF THE TIME (2)
-
○ SOME OF THE TIME (3)
-
○ ALMOST NEVER (4)
-
Q4. I can correctly change a patient's tracking status (example “provider in room”) without asking someone for help.
-
○ ALWAYS (1)
-
○ MOST OF THE TIME (2)
-
○ SOME OF THE TIME (3)
-
○ ALMOST NEVER (4)
-
Q5. When I am taking a history, I can type AT LEAST HALF of the history into the HPI section while I am in the room with the patient.
-
○ ALWAYS (1)
-
○ MOST OF THE TIME (2)
-
○ SOME OF THE TIME (3)
-
○ ALMOST NEVER (4)
We have a few more questions about the NextGen EHR system. Again, there are no right or wrong answers. Your answers will help us better understand how to conduct trainings. How confident do you feel doing the following tasks in the NextGen EHR system at this point in your training?
-
Q6. Adding a new allergy to the allergy module.
-
Q7. Documenting a telephone call in the chart.
-
Q8. Writing a patient plan phrase and adding it to My-Phrases library.
-
Q9. Ordering a rapid strep test (strep test that is performed in the office)
-
Q10. Finding the results of a laboratory test recently done on the patient (example, CBC, lead).
This is the end of the survey. Many thanks for your participation!
