Keywords
electronic health record - EHR - research - resident - EHR training - primary care
- learner - assessment
Background and Significance
Background and Significance
The Electronic Health Record (EHR) has improved many aspects of patient care, including
quality, efficiency, and cost but has significantly impacted provider–patient dynamics
and clinical workflow.[1]
[2] In a recent national survey, Graduate Medical Educators agreed that the EHR impacts
the education of residents and fellows in several ways, especially in the importance
of “balancing focus on electronic documentation with patient engagement,” and the
need to teach EHR use through hands-on practice.[3] Many EHR training programs are technical, brief, and not customized to the learner's
clinical specialty. Trainer experience and learner engagement vary. In our experience,
residents do much of their EHR learning during clinical care time, adopting the habits
and workarounds of their colleagues. As resident physicians are central to academic
institutions, suboptimal EHR training can result in inefficiency and user frustration
and the need for significant ongoing support.[4]
[5]
[6] Such delays interrupt patient care, increase workload, and residents have reported
decreased available education time.[4] This is worrisome, as command of information technology is “vital in the modern
practice of medicine.”[7]
Effective EHR learning programs incorporate social cognitive theory and adult learning
principles.[8]
[9] Several studies suggest that enhanced EHR training positively impacts EHR adoption,
efficiency, and patient care.[5]
[10] Enhanced EHR training has been investigated for large groups of residents across
multiple specialties[4]
[11]
[12] and medical students,[13]
[14] but objective measurement of these interventions is absent or in early phases of
exploration.[11]
[15]
[16] Most published literature on this topic only utilizes self-assessment methods.
Objectives
-
To design and implement an enhanced EHR learning curriculum that incorporates adult
learning theory and multimodal educational tools relevant to the PGY1 resident.
-
To create and implement assessment tools for EHR proficiency and self-reported efficiency
and self-efficacy.
Methods
This study was completed at a large academic pediatric practice. All new PGY1 residents
received basic training taught by a nonclinical software specialist. Basic training
occurred in a classroom setting with individual desktop computers and a large instructional
monitor. Content was not specialty specific and was presented in a didactic fashion
with occasional individual learner task practice. In the months following initial
go-live in January 2012, EHR superusers conducted an informal needs assessment in
the resident clinic. Superusers were attending physicians who had received additional
EHR training, led site implementation and clinical support, and were regarded as local
experts. Superusers noted that in the months after EHR adoption, preceptors were devoting
approximately 30 to 60 minutes per clinic session to instructing residents in EHR
use and correcting EHR errors, instead of providing clinical education. Superusers
also noted widespread deficiencies in residents' knowledge of basic EHR tasks and
decreased time spent on clinical care. In response to the needs assessment, superusers
created an enhanced EHR training curriculum for incoming residents built upon social
cognitive theory and adult learning principles frequently used in graduate medical
education (see Appendix 1).
Our study was a prospective, single center cohort study of Pediatric PGY1s' EHR skills
and attitudes before and after the enhanced EHR training curriculum. All new PGY1s
were required to participate in the basic and enhanced training, and all were invited
to participate in this study from the start. Written consent was obtained for those
who agreed. The EHR utilized in our practice was NextGen Ambulatory EHR (QSI Management,
LLC; Horsham, Pennsylvania, United States).
Enhanced Training Intervention
We designed an enhanced, three-phase EHR training curriculum to follow basic training
(Appendix 2).
Phase 1 was designed to orient PGY1s to EHR layout and basic visit workflows. An EHR
physician superuser led small group classes over 4 hours, reviewing a typical pediatric
visit using demonstrations, learner practice activities and role plays, and entry
of medications and vaccines. For example, a volunteer pretended to be a parent, using
a script to give the history for a sick child, while PGY1 learners documented within
the Histories template. In another example, PGY1s were provided a list of 10 frequently
prescribed pediatric medications (such as amoxicillin suspension) and practiced creating
prescriptions within the medication module. All EHR terminology was defined in language
familiar to learners.
Phase 2 was held within the PGY1's continuity clinic to situate an EHR practice experience
within the actual clinical setting. Residents were provided 60 minutes during the
first week's orientation to strengthen and reinforce EHR skills through role playing
with data entry. For example, a pair of PGY1s would use an exam room to role play
as patient and physician, with the physician entering data into the Histories, Review
of Systems, Physical Exam, and Assessments templates. PGY1s also utilized this time
to practice creating prescriptions that were then reviewed by an attending physician,
and later set up saved “favorite” medications, My-Phrases, and templates within the
system.
Phase 3 was held during the PGY1s' second clinic week, focusing on skill application
with supervision and feedback. As real-time, one-on-one EHR training in the work environment
has been shown to enhance performance,[6] PGY1s observed two patient care visits performed by their senior residents (PGY3)
and then acted as scribes for subsequent visits. As PGY3 residents had been using
the system for 2 years, they served as experienced peer educators to support new learners'
skill acquisition during this session.[8] PGY1 residents also entered orders and completed electronic forms under supervision
during the PGY3 precepting time.
Assessment of Residents
EHR skills were measured through a proficiency assessment tool, and surveys measured
self-reported EHR efficiency and self-efficacy. The proficiency assessment and surveys
were administered together on two separate occasions: following basic training (pretest)
and following enhanced training (posttest, [Fig. 1]).
Fig. 1 Schematic of curriculum and evaluation.
Proficiency Assessment Tool
To our knowledge, there exists no validated, widely used measure of EHR proficiency.
To create our own measure and establish that its content represented the appropriate
skills,[17] six typical EHR tasks for pediatric primary care were assigned to experienced clinicians
for evaluation ( Appendix 3) and adjusted so that they could be completed within 6 minutes. Each task was given
a score of 0 (not done), 1 (partially done), or 2 (completely done) and equally weighted
toward a total score from 0 to 12. Each task was preassigned a detailed definition
of the three possible scores, including attention to quality of content. At the pre-
and posttests, each PGY1 was provided 6 minutes to complete the six tasks. Results
were scored by a clinician blinded to curriculum content and study objectives. Mean
group scores were utilized for analysis.
Efficiency and Self-Efficacy Surveys
Since no single, validated, widely used tool existed to measure these data, question
themes were culled from the literature. Questions were piloted with physicians in
the practice (details in Appendix 3). Five efficiency questions used the stem, “I can correctly complete [EHR task] without
asking someone for help...” with a 4-option Likert response scale. Five self-efficacy
questions used the stem, “How confident do you feel doing the following tasks...?”
with a 4-option Likert response scale.
Analysis
For pre- and postcurriculum results, paired t-tests were employed to compare mean proficiency assessment scores, and the Wilcoxon
Signed Rank Test (WSRT) compared the survey differences. Only assessments with matching
pre- and postresponses were included. Survey data were imported into SPSS for analysis.
Assuming 25 participants, we estimated 80% power to detect a change of ≥ 1.5 points
on each scale with an estimated standard deviation of 2.5 and an α of 0.05 using a 2-sided one sample t-test.
Results
The incoming class of 26 PGY1s initiated the basic and enhanced training and consented
to study participation. One PGY1 could not complete the training due to illness, and
six individuals did not complete both pre- and posttests. This resulted in a sample
size of 19 of 26 or 73%.
Proficiency Assessment
The mean total proficiency scores improved with statistical significance from pre-
to posttesting (6.8–7.8, p = 0.048; [Fig. 2]). Of the six items, there was statistically significant improvement in medication
entry (mean: 1.0–1.7, p = 0.02) and completing school forms (mean: 0.2–0.7, p = 0.03) with nonsignificant changes in the remaining items (adding vital signs, adding
allergies, adding diagnoses, and ordering vaccinations).
Fig. 2 Mean proficiency assessment scores pre- and postcurriculum for six tasks and total
for all tasks. *p < 0.05.
Efficiency and Self-Efficacy
Residents reported a low sense of efficiency and self-efficacy on the pretest. For
pretest efficiency, only a minority of residents reported that they could correctly
complete tasks “always” or “most of the time” ([Fig. 3]). For pretest self-efficacy, a minority of residents felt “very confident” or “somewhat
confident” in three of the tasks, although they reported confidence while adding allergies
or documenting a telephone call ([Fig. 4]). In the posttest, most residents reported a higher sense of efficiency than on
the pretest, including vaccine ordering (WSRT, p < 0.05), complete blood count (CBC
ordering; WSRT, p < 0.05), school physical form completion (WSRT, p < 0.05), and updating patient location (WSRT, p < 0.05; [Fig. 3]). History taking did not improve significantly (WSRT, p = 0.074). Residents reported more self-efficacy on the posttest for adding saved
phrases (WSRT, p < 0.05), but the remainder reported less or stayed the same, including documenting
a telephone call (WSRT, p < 0.05), adding allergies (WSRT, p = 1.00), ordering a rapid strep test (WSRT, p = 0.49), and locating test results (WSRT, p = 0.22; [Fig. 4]).
Fig. 3 Efficiency survey results pre- and post-curriculum for five tasks. Question stems,
“I can correctly complete (task) without asking someone for help…” *p < 0.05.
Fig. 4 Self-efficacy survey results pre- and postcurriculum for five tasks. Question stems,
“How confident do you feel doing the following tasks?” *p < 0.05.
There were similarities between results for those activities that were measured by
both the surveys and the proficiency test. For updating allergies, most respondents
reported confidence and had high proficiency scores at both the pre- and postassessments
([Figs. 2] and [4]). For ordering vaccines, respondents reported very low efficiency at the pretest
and made modest improvements on the posttest ([Fig. 3]). In a similar pattern, proficiency scores were low and did not improve ([Fig. 2]). For school form completion, residents first expressed low efficiency and had low
proficiency scores, and both measures significantly improved from pre- to postassessments
([Figs. 2] and [3]).
Discussion
As suspected, participating PGY1 residents did not report substantial efficiency or
self-efficacy after basic EHR training. To develop an enhanced curriculum for the
use after basic training, we incorporated additional adult learning theory, simulation,
physician trainers, and one-on-one experiences.
These results suggest that a three-phased enhanced training curriculum may have improved
resident EHR efficiency and some measures of proficiency. These findings are relevant
given that EHR competence is now a basic clinical skill, and residents who lack proficiency
and efficiency may struggle to perform clinical tasks. Although medicine and medical
education are highly regulated fields, the literature about measuring EHR skill acquisition
is scarce. This article adds to the field by describing a novel way to assess EHR
skill acquisition. This study could contribute to future development of standardized
training and objective assessment of EHR provider–users across multiple primary care
settings. Of note, the biggest resource necessary for our intervention was physician
time (4 × 2 = 8 hours), as EHR superuser attending physicians delivered the curriculum.
This time resource, however, was less than the estimated amount of time required for
on-the-spot EHR assistance provided by attending physicians for residents prior to
this study.
Certain outcome measures showed no improvement or worsened. Although two individual
proficiency scores and the average proficiency scores showed improvement, several
scores worsened, which may reflect differing complexity of particular tasks. Additionally,
the three of the four tasks that showed no improvement in proficiency were relatively
simple tasks (vital signs, allergies, and diagnosis) that had high initial scores.
It may be that learners achieved mastery after basic training and therefore had little
room to improve further. Worsening self-efficacy may be due to the complex user interface
of the EHR application. Also, as large parts of phases 2 and 3 were peer-to-peer,
the quality of the training interaction was not established. In addition, it is possible
that experienced users in phase 3 passed on maladaptive workarounds, worsening learner
scores. Lastly, residents also may reach a “saturation point” of new information during
a short but demanding orientation period.
Related studies published on this topic have mixed results. Dastagir and colleagues
surveyed clinicians already experienced in their EHR who received a proficiency training
course. Comparing pre- and posttraining surveys, they demonstrated improvements in
self-assessed efficiency.[5] Vuk and colleagues demonstrated improved self-efficacy assessed before and after
simulation training,[10] which differed from our self-efficacy results. Jalota and colleagues conducted a
real-time peer training intervention with physicians already experienced in their
EHR and showed improved documentation efficiency over the control group.[6] Reis and colleagues demonstrated no improvement in attitude, sense of competence,
or observed proficiency when comparing a simulation based training to traditional
lecture format.[12]
Our study had several limitations. Self-report of efficiency might not have reflected
actual efficiency. Our sample size was small, and performance at a single center might
have limited its external validity. A lack of a control group limited our ability
to attribute changes to the intervention instead of increased training exposure alone
(maturity threat). A selection bias might have been introduced by including only new
residents, who typically were younger and more comfortable with technology than other
physicians. The majority of our assessments dealt with data entry tasks, even though
data extraction and interpretation were important EHR skills. Although our curriculum
was adapted to our EHR application, it could be easily adapted to other systems.
Single group pre-and posttest models, such as ours, are at a risk of testing threat,
in which exposure to the pretest influences participants' learning, thereby exaggerating
their performance on the posttest. Our proficiency assessment and surveys evaluated
slightly different sets of EHR tasks, therefore evaluating the exact set of tasks
through the lens of both the surveys and the proficiency tool might have allowed the
curriculum to be refined and focused on tasks requiring more instruction to achieve
mastery. The assessment tools also deserve more thorough validation, including against
clinical outcomes. However, this study was an attempt to integrate adult learning
principles into EHR training and describe a method of EHR skill assessment in a manner
that could be replicated. Study leaders received verbal feedback from colleagues describing
stronger PGY1 EHR skills and the additional time available for precepting and clinical
education after this intervention. Future studies could formally measure this preceptor
feedback and measure time available for EHR teaching.
Next steps in this project would involve pretraining baseline assessments and further
development of the proficiency measure. Recruiting a control group at a comparable
institution, repeating the posttest at a later date, and monitoring improvements in
accuracy and efficiency during subsequent clinical care would enrich our understanding
of the effects of the intervention.
Conclusion
This study details an EHR training intervention that may improve residents' EHR skills;
it proposes several measures of EHR skill proficiency and efficiency that may be studied
further. Ultimately, an enhanced EHR training curriculum should strengthen residents'
EHR skills and decrease interruptions, allowing residents and preceptors to prioritize
direct patient care and build clinical knowledge.
Clinical Relevance Statement
Clinical Relevance Statement
Residents and EHRs build the backbone of academic medical institutions; however, resident
EHR training is often overlooked. This study presents a novel, enhanced EHR training
intervention designed with adult learning principles, and measures learners' proficiency,
efficiency, and self-efficacy. Significant improvements in efficiency and some measure
of proficiency are noted, which could support streamlined workflows, patient care
quality, and time for clinical education.
Multiple Choice Questions
Multiple Choice Questions
-
Training curricula for clinicians to use an EHR should include
-
A fast-paced style to keep the learner's attention
-
Careful attention to the concerns about the application raised by the learners
-
Well-trained mock patients
-
Active learning techniques using real-life scenarios
-
When compared with standard EHR training, enhanced EHR training
-
Can positively impact the efficiency of clinicians' EHR use
-
Incorporates adult learning principles and/or social cognitive theory
-
Can require more time to complete
-
All of the above
Answers
-
Correct answer is D, Active learning techniques using real-life scenarios. Adult learning theory uses real-life scenarios and active engagement of the learner.
The quality of the mock patient, being fast paced to keep the learner's attention
and time spent on learners' concerns of the EHR application design are less important
components of EHR training for medical providers.
-
Correct answer is D, All of the above. Incorporation of adult learning principles can result in a longer training period
when compared with out-of-the-box training. However, evidence suggests that this could
result in increased EHR user efficiency.
Description of Need Assessment and Curriculum Creation
As an informal needs assessment, EHR superusers spoke with a majority of attending
preceptors inquiring about residents' proficiency or lack of proficiency with the
various EHR workflows, types of errors made by residents in the EHR system, and what
fraction of attending physicians' time was spent teaching the EHR instead of teaching
clinical skills. Preceptors responded that residents were unfamiliar with the documentation
of procedures, had difficulty remembering how to e-prescribe medications, and were
not familiar with ordering point-of-care tests or utilizing the form generator, among
other challenges. Additionally, 2 months after the initial go live, superusers, who
also served as regular clinic preceptors, noted approximately 30 to 60 minutes per
half-day clinic session devoted by preceptors to EHR instruction instead of clinical
teaching about routine pediatric care, disease, or clinical skills. Superusers received
similar, although informal, comments from colleagues about this matter. Superusers
also spoke with one or two residents during each “clinic day” (Monday–Friday) regarding
components of the EHR in which they felt that they needed additional training. Residents
responded that they felt a requirement for additional training on how to order point-of-care
and send out laboratory tests, document telephone calls, e-prescribe complex medications,
capture history in real time, and generate commonly used forms. Additionally, residents
reported having to physically go and find a staff person or colleague if they had
a question about the EHR, which was time consuming.
Superusers reviewed an original list of all existing workflows in the current clinical
practice and ordered them chronologically in relation to when they typically occur
during patient care. Each workflow was then broken down into discrete steps, and the
steps were written into the verbal curriculum script using plain language, forming
the backbone of the enhanced curriculum. Particular attention was paid to the tasks
in which current residents had reported inadequate training.
Next, superusers reviewed tools held as best practices in adult medical education,
namely, the use of standardized patients or role playing to practice a new skill,
real-time skill quizzing with instructor feedback, opportunities for self-directed
practice, learning sessions conducted within actual clinical spaces, and the opportunity
to shadow experienced peers performing complex tasks[9]. In addition, we reviewed published work on the role of social cognitive theory
within EHR training,[10] which suggested that successful EHR training programs emphasized the positive potential
of EHRs upon patient outcomes, incorporated active learning activities, identified
positive role models for EHR use in the clinical environment, and incorporated an
understanding of the characteristics of the community within which learners may eventually
use the EHR.
In a deliberate, step-wise fashion, adult learning tools were carefully integrated
into the list of clinical workflows, with careful attention paid to task complexity
and the circumstances of real-life practice. For example, a parent/doctor role play
script was written to be performed as PGY1s included learning the Histories Template,
with instructions for learners to watch the actors and document the history of present
illness (HPI). A practice set of 10 prescriptions was written to be used for independent
practice prior to instructor feedback. Superusers created rhymes and humorous phrases
as memory aids for tasks with multiple steps. The curriculum was then divided into
phases (see Appendix 2) with opportunities for advanced practice of key workflows
provided in a three-phase curriculum setup. Finally, introductory material was created
to welcome learners, place the EHR within the context of patient care and their graduate
education, and identify individuals in the clinical setting who could provide support
during their EHR use.
Detailed Description of Enhanced Training, Phases 1, 2, and 3
Phase 1 of enhanced training was designed to orient PGY1s to the EHR layout and the
basic workflows of ambulatory pediatric visits. Two classes (14 residents each) ran
concurrently in identical training classrooms during the second week of residency
orientation. Phase 1 instruction was led by general pediatrics and EHR physician superusers,
who had advanced EHR experience and training. Both instructors trained from identical
scripts to minimize variations. PGY1s were instructed to ask the instructors questions
at any time and notify the teacher in the event of confusion or difficulty.
The phase 1 training followed the typical sequence of events of a well-child visit.
The “visit” was divided into 15 sections (reason for visit/chief complaint, vital
signs module, social history, physical exam, assessments, ordering laboratories, plan
details, referrals, immunization modules, anticipatory guidance, procedure module,
confidential teen history, medication module, document library, and laboratory results).
For each section, the teacher would introduce the section of the visit and its purpose,
demonstrate how to navigate to the appropriate area in the EHR, and demonstrate three
times how to document in or use the functions of that section. The teacher would then
instruct students to navigate to the section and have them perform a particular documentation
or function within that section. Other smaller sections were covered if time allowed.
Additional exercises were given to reinforce skills in certain sections. For example,
in the Social History section, volunteers used scripts to role play as parents while
being interviewed about the family's home circumstances, and students typed the details
into the Social History section. For the medication module, PGY1s were provided five
pediatric medication examples to enter into the module. For the confidential teen
history, volunteers role played as teens while being interviewed about their risk-taking
behaviors, and PGY1s typed their data into the teen section. PGY1s were also provided
a short list of immunizations to order over a 5-minute period. Additional teaching
tools included props, humor, and short rhymes to help with memory.
Phase 2 occurred 1 to 4 weeks after phase 1 and was designed to situate an EHR practice
experience within the actual clinical setting. This phase took place within the ambulatory
primary care clinical space where PGY1s have their weekly continuity clinic. As part
of the traditional residency orientation, PGY1s spent two dedicated afternoons in
this space to meet the staff and had a tour, and phase 2 occurred on the first of
these days. PGY1s were provided a workbook detailing their assignments. For the first
assignment (40 minutes), residents formed groups of two and were provided two scripts
of typical pediatric patient histories. They were instructed to simulate a typical
clinical encounter by using an empty exam room and logging into the EHR system, where
a “dummy” chart had been created for training purposes. With one resident reading
from the parent script, the other resident would role play as the physician, and ascertain
the patient's medical history while entering it into the EHR system. They would then
repeat this exercise using the second script and reversing the roles. Attendings and
upper-year residents familiar with the system were available in the event of questions
or problems during the exercise.
For their second assignment (10 minutes), residents were provided a list of seven
common medications with specific dosages and instructed to create the prescriptions
within the EHR, print them, and have the prescriptions reviewed by an attending physician
for accuracy and feedback. Their final assignment (10 minutes) was to set up and save
their preferred “favorite” settings in the system, for example, designating frequently
used medications in an easy-to-find folder.
Phase 3 of enhanced training allowed PGY1s to apply their EHR skills with close supervision
and feedback. This phase occurred 1 to 4 weeks after phase 2. For the first 2 hours,
PGY1s shadowed an upper-year resident during two routine visits. During the third
visit, PGY1s scribed the history into the EHR in real time while the upper-year resident
conducted the visit. During the precepting session with the attending physician, the
PGY1 was closely supervised by the attending physician while updating the in-room
patient tracking status and inputting necessary orders for the patient (for example,
vaccines, laboratory tests, and referrals). The resident was also instructed to populate
and generate the patient's school physical forms in the EHR. Immediate feedback was
provided to the PGY1 by the attending physician on form accuracy and order correctness.
Residents used any remaining time to set up custom “favorites” within the system and
practice making common patient health documents.
Proficiency Assessment and Surveys
Proficiency Assessment Tasks*:
(*Pretest tasks. Posttest tasks were similar but used different pulse value, allergy,
diagnosis, etc.)
-
Your patient's pulse is 111 beats/minute. Add this heart rate to the vital signs template
and save it.
-
Your patient mentions a peanut allergy. Add this allergy to the allergy module. Indicate
that the allergy is severe.
-
You diagnose your patient with ASTHMA. Add this to today's assessments.
-
In the medication module, prescribe albuterol HFA, 2 puffs q4h PRN cough and print
the prescription.
-
Your patient requests a child health assessment. Create a child health assessment.
Indicate the asthma history and the peanut allergy in the appropriate sections. Also,
in the appropriate section, type that she/he needs to have albuterol and an EpiPen
readily available.
-
Your patient needs a hepatitis A vaccine. Order the HepA vaccine and task to the appropriate
recipient.
Survey
The following questions ask about your experience completing tasks in the NextGen
EHR system. Choose the answer that best represents your experience at this point in
your training.
-
Q1. I can correctly order a vaccine without asking someone for help.
-
○ ALWAYS (1)
-
○ MOST OF THE TIME (2)
-
○ SOME OF THE TIME (3)
-
○ ALMOST NEVER (4)
-
Q2. I can correctly order a CBC without asking someone for help.
-
○ ALWAYS (1)
-
○ MOST OF THE TIME (2)
-
○ SOME OF THE TIME (3)
-
○ ALMOST NEVER (4)
-
Q3. I can correctly create a child health assessment without asking someone for help.
-
○ ALWAYS (1)
-
○ MOST OF THE TIME (2)
-
○ SOME OF THE TIME (3)
-
○ ALMOST NEVER (4)
-
Q4. I can correctly change a patient's tracking status (example “provider in room”)
without asking someone for help.
-
○ ALWAYS (1)
-
○ MOST OF THE TIME (2)
-
○ SOME OF THE TIME (3)
-
○ ALMOST NEVER (4)
-
Q5. When I am taking a history, I can type AT LEAST HALF of the history into the HPI
section while I am in the room with the patient.
-
○ ALWAYS (1)
-
○ MOST OF THE TIME (2)
-
○ SOME OF THE TIME (3)
-
○ ALMOST NEVER (4)
We have a few more questions about the NextGen EHR system. Again, there are no right
or wrong answers. Your answers will help us better understand how to conduct trainings.
How confident do you feel doing the following tasks in the NextGen EHR system at this
point in your training?
-
Q6. Adding a new allergy to the allergy module.
-
Q7. Documenting a telephone call in the chart.
-
Q8. Writing a patient plan phrase and adding it to My-Phrases library.
-
Q9. Ordering a rapid strep test (strep test that is performed in the office)
-
Q10. Finding the results of a laboratory test recently done on the patient (example,
CBC, lead).
This is the end of the survey. Many thanks for your participation!
