Effective Design, Development, and Evaluation of Video Tutorials for Electronic Medical Record Training

Keywords

electronic health records - computer user training - physicians - primary care - meaningful use - video tutorials

Background and Significance

An electronic medical record (EMR)[1] can facilitate improved capture, organization, and presentation of patient information. One of the main documented benefits of EMRs is preventive care and chronic disease management (CDM). These benefits are only realized if EMRs are used effectively, which includes complete, accurate, and consistent documentation, and use of advanced features such as recalls/reminders. However, current EMR use by primary care physicians (PCP) in the United States and Canada is suboptimal, especially for supporting chronic diseases[1] [2] [3] [4] [5] [6] [7] like diabetes. PCPs need support and training to achieve this kind of value-adding use[1] [8] [9] [10] [11] [12]. In particular, most PCPs require postimplementation training, as this is an often neglected area of training and probably the most needed by practicing clinicians.

Training provides “the conceptual and procedural knowledge necessary to put the technology to effective use”[13]. Common methods of software training include: (1) course/lecture/seminar; external training, computer-aided instruction, resident expert, and tutorials.[14] In the past two decades, video tutorials have become increasingly popular method for providing software training.

A video tutorial demonstrates how to accomplish tasks using software,[15] and it is a simple, affordable tool to produce authentic, situated, and motivational instructional material.[16] The benefits of video tutorials include: (1) support for users' development of a mental model of the software[16], (2) faster initial learning and better comfort compared with static versions of instructions[16], and (3) increased user control and autonomy.[16] However, a limitation of video tutorials is the “mimicry model” (i.e., memorizing and copying steps without internalizing the task).[17] [18] [19] Another disadvantage is the lack of an inferential step, potentially leading to lower retention of information compared with written instruction. Regardless, video tutorials can reduce cognitive processing for end-users, allowing them to immediately practice the demonstrated skills.[16] Video tutorials may also be considered a cost-effective and scalable tool for vendors to provide software training.

To support electronic health record (EHR) training for nursing students, He et al (2016) had designed and used two EHR video tutorials.[20] The authors identified that video tutorial-based training programs can be improved through hands-on practice and review questions with answers.[20] In another study, Thiyagarajan et al used EHR video tutorials to provide EHR training to medical students.[21] The medical students provided informal positive feedback about the video tutorials. The authors also reported that the video tutorials helped decrease the amount of time needed for EHR staff to orient student clinicians to the EHR.[21] Similarly, Zoghbi et al[22] provided EHR video tutorials to a group of general surgery residents. The study reported statistically significant positive effects of the video tutorials on residents' confidence in carrying out EMR tasks and clinical scores on emergency simulations. The video tutorials also helped to decrease the residents' time required to perform essential EMR tasks.[22] Recently, Randhawa et al (the authors) published a small-scale efficacy study that demonstrated the potential of video tutorials to improve EMR use for chronic diseases, such as diabetes.[23] However, primary research studies on the design of video tutorials for EMR training of PCPs or other practicing physicians have not been published to date.

Currently, there is a considerable need to design EMR theory-informed and evidence-based video tutorials for PCPs. Further, the EMR video tutorials should be designed to support learning for value-adding EMR use. The purpose of the study was to address this gap through the design, development, and evaluation of video tutorials for training PCPs in using EMR advanced features for diabetes care.

Introduction

The effective design of video tutorials should draw from multiple learning and instructional design theories and principles including usability,[24] minimalism,[25] structured writing (information mapping),[26] multimedia learning and multimedia principles,[27] animation,[28] attention,[29] dual coding theory,[30] social learning theory,[31] [32] the four components model,[33] guidelines for recorded demonstrations,[34] and demonstration-based training model[35] (which is based on social learning theory). Key principles from this body of literature are outlined in [Table 1]. In this paper, we focus on social learning theory and dual coding theory.

Table 1 Key principles from learning and instructional design theories/principles

Theory/principle	Key principles
Usability	• Ease of use of information (i.e., easy to find, easy to understand, and task-sufficient or sufficient for the task at hand).[24]
Minimalism	• Choose an action-oriented approach, anchor the tool in the task domain, support error recognition and recovery, support reading to do, study, and locate.[25]
Structured writing (information mapping)	• Create and systematically organize independent units of information to ensure that they are accessible, manageable, and reusable.[26]
Multimedia learning and multimedia principles	• The human information processing system includes dual channels for visual/pictorial and auditory/verbal processing.[27] • Each channel has limited capacity for processing.[27] • Active learning involves carrying out a coordinated set of cognitive processes during learning.[27]
Animation	• Animated graphics should be effective in portraying change over time.[28]
Attention	• Humans have limited capacity.[29]
Dual coding theory	• Visual images, coupled with verbal information, aid learning.[30]
Social learning theory	• Human learning by observing and imitating others.[31] [32]
Four components model	• A procedure consists of goals, prerequisites, subgoals, interim states or subgoals, unwanted states, and actions and reactions.[33]
Guidelines for recorded demonstrations	• Technical guidelines: keep file sizes small, strive for universal usability, and ensure user control, etc. and provide tips to achieve those goals.[34] • Content guidelines include: create short demonstrations that focus on tasks, highlight each step with auditory and visual cues, synchronize narration and animation carefully, and create demonstrations with a clear beginning, middle, and end.[34]
Demonstration-based training model	• As an extension/application of social learning theory, instructional features in design should address the four interrelated processes of: attention, retention, production, and motivation.[35]

Based on the theories in [Table 1] and empirical studies of video tutorials, several best practice guidelines have been proposed over the years[34] [36] [37] for designing software video tutorials. However, the eight recommendations developed by van der Meij and van der Meij[36] in 2013 are the most comprehensive set, and “summarize key notions of accepted thinking” for designing instructional videos.[36] These guidelines are[37]:

1. Guideline 1: provide easy access

   • A. Guideline 1.1: craft the title carefully

2. Guideline 2: use animation with narration

   • A. Guideline 2.1: be faithful to the actual interface in the animation

   • B. Guideline 2.2: use a spoken human voice for the narration

   • C. Guideline 2.3: action and voice must be in synch

3. Guideline 3: enable function interactivity

   • A. Guideline 3.1: pace the video carefully

   • B. Guideline 3.2: enable user control

4. Guideline 4: preview the task

   • A. Guideline 4.1: promote the goal

   • B. Guideline 4.2: use a conversational style to enhance perceptions of task relevance

   • C. Guideline 4.3: introduce new concepts by showing their use in context

5. Guideline 5: provide procedural rather than conceptual information

6. Guideline 6: make tasks clear and simple

   • A. Guideline 6.1: follow the user's mental plan in describing an action sequence

   • B. Guideline 6.2: draw attention to the interconnection of user actions and system reactions

   • C. Guideline 6.3: use highlighting to signal screen objects or locations

7. Guideline 7: keep videos short

8. Guideline 8: strengthen demonstration with practice.

Another principle that informed our video tutorials design was homophily. In the end-user support literature for health information systems, homophily is “the degree of similarity between the support source and the end-user, and the degree to which the support source demonstrates understanding of the day-to-day work of the user”[38]. Homophily corresponds with modeling in social learning theory, in which learning occurs by observing a modeled behavior (e.g., physician trainer performing a task using EMR in a video tutorial) and then practicing or reproducing that behavior.[31] [32] [39] [40] It has been shown that adoption of the modeled behavior by individuals is higher when the model is similar to the individual(s).[32] Homophily is also one of the three essentials for learning, as suggested by Eyal and Rubin,[41] along with identification (ability of the observer to engage in perspective taking and share in the observee's experience) and parasocial interaction (friendship or bond with the observee).

This paper describes the implementation of the van der Meij and van der Meij[37] guidelines in video tutorials for using EMR advanced features for diabetes care, and the perceptions of PCPs of these videos.

Research Questions

This study addressed the following research questions:

1. What are PCP's views of video tutorials as a training method/approach for learning EMR advanced features?

2. What are the barriers and facilitators to applying the EMR video tutorials-based training to PCPs' practice?

3. How can the design of EMR video tutorials be improved?

Methods

The overall research study employed a QUAN (qual) mixed methods approach with an embedded design. In this paper, we report the findings from the qualitative portion of the study, which focused on the video tutorial design. Ethics approval was obtained from the University of Victoria Human Research Ethics Board (ethics protocol 17–189). The quantitative results of the mixed methods study (i.e., efficacy of the video tutorial intervention) are reported in a companion paper.[23]

Setting and Sample

Sampling was carried out as a part of the mixed methods study. Study participants included PCPs who use OSCAR EMR, which is one of the major EMRs used in British Columbia (BC) Province, Canada. All PCPs who use the OSCAR EMR in BC (n = 984) were invited to participate in the study. The BC Divisions of Family Practice and OSCAR BC Users' Group recruited participants using an invitation letter and a YouTube video[42] that explained the research study. The Divisions of Family Practice advertised the study opportunity to their division members through emails, newsletters, division meetings, and word of mouth. Interested PCPs signed and submitted a consent form to participate in the study, and were included on a “first come, first served” basis.

Intervention

The study intervention, called MD-PET (Management of Diabetes Postimplementation EMR Training), included a series of four short online video tutorials, each targeting one of the following EMR features for diabetes care:

1. Create and maintain a diabetes registry by coding diabetic patients using the “250” International Classification of Diseases, Ninth Revision (ICD-9) code for diabetes mellitus.

2. Create recalls/reminders for diabetes care.

3. Order/view hemoglobin A1C in the EMR.

4. Record blood pressure for diabetes care visits in the EMR.

These EMR features are similar to the EMR use instructions and applications recently published by Singer and Ivers.[43] The Chronic Care Model (CCM) served as the conceptual framework that underpinned the EMR training intervention, as it is “the best known and most influential” organizational model for chronic care”[44]. The CCM aims to develop well-informed, activated patients interacting with a practice team that is proactive and prepared for them; its end-goal is to improve health outcomes. The video tutorial intervention focused on the “clinical information systems” and “delivery systems design” components of the CCM (i.e., developing and maintaining, registries, and reminders/recalls). The content and topic selection for the video tutorials was determined based on the BC Clinical Guidelines for diabetes care. The order of the video tutorials was based on proactively planning a follow-up diabetes care visit in the EMR for a patient, and aligns with the CCM in terms of developing a patient registry, recalling patients, and viewing and recording process measures for diabetes care.

The guidelines from van der Meij and van der Meij were implemented in designing the video tutorials as described in [Table 2]. In total, 12 guidelines and subguidelines were fully applied and three were partially applied (i.e., guidelines 4.1, 7, and 8). Although users were provided the learning goal for every video as per guideline 4.1 (promote the goal), they were not provided a preview of the task. While guideline 7 (keep videos short) recommends that videos on medical consultation for problem-based learning be kept to 3 minutes, the length of the video tutorials in the study intervention ranged from 3 minutes and 48 seconds to 4 minutes and 48 seconds. Additionally, to ensure that users watched all the video tutorials in the study intervention, users were instructed to practice the tasks at the end of the final video tutorial only. Consequently, guideline 8 (strengthen demonstration with practice) was not fully met.

Table 2 Application of video tutorial design guidelines

Video tutorial design guideline[37]	Subguideline	Applied (yes/no)	How it was applied
Guideline 1: provide easy access	Guideline 1.1: craft the title carefully	Yes	The title of all the video tutorials contains a verb and an object to tell the user what task the video demonstrates how to perform.
Guideline 2: use animation with narration	Guideline 2.1: be faithful to the actual interface in the animation	Yes	The actual interface of OSCAR EMR was captured in the video tutorials using screencasting software.
	Guideline 2.2: use a spoken human voice for the narration	Yes	The physician champion's voice was used for narration.
	Guideline 2.3: action and voice must be in synch	Yes	The physician champion's actions and voice were in synch.
Guideline 3: enable functional interactivity	Guideline 3.1: pace the video carefully	Yes	The physician champion kept a conversational tempo and ensured that he did not speak instructions too quickly and included brief pauses.
	Guideline 3.2: enable user control	Yes	Users were able to start, pause, stop, and replay the videos.
Guideline 4: preview the task	Guideline 4.1: promote the goal	Partial	Users were provided the learning goal for every video. However, users were not provided a preview of the task.
	Guideline 4.2: use a conversational style to enhance perceptions of task relevance	Yes	The physician champion delivered instructional messages in a conversational style.
	Guideline 4.3: introduce new concepts by showing their use in context	Yes	The video tutorials introduce new-related concepts, as needed (e.g., rationale for color coding of measurements after showing users how to record blood pressure).
Guideline 5: provide procedural rather than conceptual information	N/A	Yes	Presenting procedural (how-to) information was the focus of the video tutorials. Conceptual information was kept to a minimum.
Guideline 6: make tasks clear and simple	Guideline 6.1: follow the user's mental plan in describing an action sequence	Yes	The video tutorials follow the sequence in which users generally engage in EMR task execution for diabetes care visits.
	Guideline 6.2: draw attention to the interconnection of user actions and system reactions	Yes	The video tutorials state the goal or purpose, as well as demonstrate how (i.e., the actions) to achieve the goal.
	Guideline 6.3: use highlighting to guide attention	Yes	Highlighting of the mouse was used in the video tutorials.
Guideline 7: keep videos short	N/A	Partial	The length of the video tutorials ranged from 3 minutes and 48 seconds to 4 minutes and 48 seconds. They were slightly longer than the suggested 3-minute average for videos on medical consultation for problem-based learning.[37]
Guideline 8: strengthen demonstration with practice	N/A	Partial	To ensure that users watched all the video tutorials in the study intervention, users were instructed to practice the tasks at the end of the final video tutorial only.

Consistent with the concept of homophily discussed earlier, a physician champion was involved in the design and recording of the video tutorials. The same physician champion recorded all of the video tutorials. In addition to the computer (EMR) screen, the physician champion's face was recorded simultaneously. The video tutorials were recorded, edited, and produced using Camtasia screencasting software, including the addition of animations, mouse highlighting, transitions, music, etc. Access to the finalized video tutorials was made available to study participants via a private YouTube video link[45] to view at their convenience during the intervention period.

Data Collection

Qualitative data were collected at four time points between July 2017 and May 2018: 1 month before (O1), immediately before (O2), 3 months after (O3), and 6 months (O4) after the intervention. This timing coincided with the quantitative data collection time points of the larger mixed methods study.[23] Data were collected using an open-ended question (“please feel free to share any comments regarding your use of the EMR for diabetes care management”) included within a quantitative Diabetes Care Questionnaire (DCQ) at O1–O4. Semistructured interviews with participants were held at O3 and O4 to understand (1) PCPs' views of the video tutorials, (2) barriers and facilitators to applying the video tutorials to PCPs' practice, and (3) opportunities to improve the design of the video tutorials ([Tables 3] and [4]). Interviews were conducted over the phone, audio-recorded, and transcribed verbatim by the first author. The interviews were 15 to 45 minutes in duration.

Table 3 OSCAR electronic medical records interview questions (O3)

1. Which EMR features from the training did you choose to use and why? 2. Which EMR features from the training did you choose not to use and why? 3. What helped with the integration of the EMR training into your diabetes care management for patients? 4. What interfered with the integration of the EMR training into your diabetes care management for patients? 5. Did you find the EMR training helpful? Why or why not? 6. Was something missing in the EMR training? 7. Is there something you could use to aid your decision-making for diabetes care management? 8. Did anything surprise you about the EMR training? 9. What role can the EMR play in helping to improve process measures for diabetes care management? 10. What role can the EMR play in helping to improve clinical indicators for diabetes care management? 11. Do you have ideas for improving the EMR training? 12. What advice would you give to someone in a similar situation (i.e., a physician interested in improving diabetes care management for their patients using their EMR)?

Abbreviation: EMR, electronic medical record.

Table 4 OSCAR electronic medical records interview questions (O4)

Thank you again for participating in the interview! I have a series of open-ended questions. May I ask your permission to record the interview? 1. You participated in the 3-month follow-up interview. What has changed since the last interview? 2. What has interfered with integrating the EMR training into your practice (i.e., barriers)? 3. What has helped with integrating the EMR training into your practice (i.e., facilitators)? 4. Have you taken any additional steps to improve your EMR use in general? What about diabetes care management in particular?  • Probe: additional training, individual trial and error, other forms of support (e.g., impersonal). 5. Are there any other forms of end-user support that you would find helpful to supplement the video tutorials?  • Probe: would a cue card be a useful supplement to the video tutorials? 6. What do you believe the characteristics of an effective trainer/host for EMR video tutorials are for physicians? For medical office assistants?  • Probe: knowledge, homophily, counseling, and communication skills. 7. What is the best way to reach physicians to tell them about video tutorials or to package it for them?  • Probe: would a postcard to the office describing how to get online training work? Does it need to be a 1:1 recommendation from another physician? What about professional newsletters or other organizational contacts? 8. Do you have ideas for improving the EMR training? 9. That is the last question I have for you! Do you have any other additional comments or suggestions you would like to share?

Abbreviation: EMR, electronic medical record.

Data Analysis

Data from free-text comments in the DCQ and follow-up interviews were analyzed using thematic analysis (i.e., familiarization with data, assignment of preliminary codes, searching for patterns/themes, review, and finalization of themes). Data were reviewed and coded in NVivo 10 qualitative data analysis software.[46] To establish trustworthiness and verify the coding, three (14%) of the interviews and two (10%) of the qualitative comments from the DCQ were double coded. To support verification of the codes, the second researcher was provided the study code book and encouraged to add additional codes as needed. The second researcher coded the data at the highest node level, and there was moderately high agreement (77%) between the coders. Disagreements were related to detailed codes that were not used by the second researcher. To resolve discrepancies and reach consensus, the more detailed code was used. The final coding scheme that emerged is available in [Table 5].

Table 5 Qualitative data codebook

Code	Subcode	Description
Additional comments and suggestions	N/A	Maps to the O3 interview that was asked at the end of the interview: do you have any addition comments or suggestions you would like to share?
Advice to physicians	N/A	Maps to O3 interview question 12: what advice would you give to someone in a similar situation (i.e., a physician interested in improving diabetes care management for their patients using their EMR)?
Decision-making aids for diabetes care management	N/A	Maps to O3 interview question 7: is there something you could use to aid your decision-making for diabetes care management?
EMR features	10 subcodes, as outlined below.	This node includes responses from: (1) O3 interviews related to interview question 1 (which EMR features from the training did you choose to use and why?) and 2 (which EMR features from the training did you choose not to use and why?), (2) O4 interview question 1 (you participated in the 3-month follow-up interview. What has changed since the last interview?), and (c) open-ended question in the O1-O4 DCM questionnaire.
	Awareness of EMR features	The PCP's awareness of the EMR features for diabetes care prior to the training intervention.
	Changes since O3	Maps to O4 interview question 1: you participated in the 3-month follow-up interview. What has changed since the last interview?
	EMR features already used prior to training	EMR features that the physician was using prior to the EMR training.
	EMR features not used from training	EMR features that the physician did not use from the EMR training. This maps to O3 interview question 2: which EMR features from the training did you choose not to use and why?
	EMR features not used prior to training	EMR features from the training that the physician did not use prior to the training.
	EMR features used from training	EMR features from the training that the physician has implemented into their practice. This maps to O3 interview question 1: which EMR features from the training did you choose to use and why?
	EMR features used that were not included in training	EMR features not covered in the training that the physician is using.
	Perception of EMR use for diabetes care	Includes answers about physicians' self-perception of use from the open-ended question in the O1–O4 DCM Questionnaire.
	Unused EMR features (not from video tutorials)	EMR features that the physician is using that were not covered in the EMR training.
	Useful EMR features	EMR features that the physician finds useful.
EMR issues	Three subcodes, as outlined below.	EMR issues that the physician has reported in the interviews. These issues do not relate to the EMR training.
	Additional functionality wanted	Additional EMR functionality that the physicians would like.
	EMR costs	Cost-related issues that the physician has with the EMR.
	Data issues	Data-related issues, such as lack of EMR data autopopulation.
EMR training	14 subcodes, as outlined below.	Relates to the EMR training (video tutorials) intervention that was delivered.
	Barriers	Maps to O3 interview question 4 and O4 interview question 2: what interfered with the integration of the EMR training into your diabetes care management for patients?
	Facilitators	Maps to O3 interview question 3 and O4 interview question 3: what helped with the integration of the EMR training into your diabetes care management for patients?
	Format	Terms used by physicians to describe the video tutorials, such as “short,” “accessible,” and “handy.” Maps to O3 interview question 5: did you find the EMR training helpful? Why or why not?
	Helpfulness of EMR training	Maps to O3 interview question 5: did you find the EMR training helpful? Why or why not?
	Improvement ideas	Maps to O3 interview question 11 and o4 interview question 8: do you have ideas for improving the EMR training?
	Missing components	Maps to O3 interview question 6: was something missing in the EMR training?
	Other EMR training topics of interest	Other EMR training topics that the physician is interested in learning about which were not covered in the training.
	Physician facilitator	The competencies and characteristics of the physician facilitator of the EMR training. This includes physicians' preferences about having a physician facilitator for the training.
	Potential uses	Potential use of the EMR training, such as for medical residents or locum physicians.
	Process for watching video tutorials	Comments about the process of watching the EMR video tutorials.
	Recall of EMR training	How well physicians remembered the EMR training. This code indicates that they were having challenges remembering the EMR training.
	Role in motivating increased EMR use	The role that the EMR training and research study has played in motivating increased EMR use by physicians.
	Supplements	Maps to O4 interview question 5: are there any other forms of end-user support that you would find helpful to supplement the video tutorials?
	Surprises	Maps to O3 interview question 8: did anything surprise you about the EMR training?
Medical office assistant characteristics	N/A	MOA characteristics that physicians have described in supporting/hindering diabetes care management.
Other end user support (EUS)	15 subcodes, as outlined below.	Other EUS that the PCP receives that is not included in the EMR training intervention.
	Additional steps taken since O3	Maps to O4 interview question 4: have you taken any additional steps to improve your EMR use in general? What about diabetes care management in particular?
	Children	EUS received from the PCPs' children.
	Cost	The cost of EUS.
	In clinic support from physician peers	EUS received from in-clinic physician peers.
	In-person EUS	EUS received in-person.
	MU program	EUS received as a part of the physician information technology office MU program.
	Objective data dashboard	An EMR data dashboard that shows PCPs' use of selected EMR features.
	OSCAR service provider	EUS received from the OSCAR service provider.
	OSCAR super-user support	EUS received from an OSCAR super-user.
	OSCAR support meetings	EUS received from OSCAR support meetings.
	OSCAR user community	EUS received from the OSCAR user community.
	Peer learning and mentorship group	EUS received from non-OSCAR physician learning groups.
	Physician colleagues	EUS received from physician colleagues.
	Quality improvement support	EUS received from quality improvement personnel.
	User documentation	EUS received from reviewing EMR user documentation, such as help manuals.
Participant characteristics	N/A	The characteristics of the participants that have emerged from the interviews through self-report.
Primary care reform	N/A	Comments related to primary care reform that emerged from the O3 interviews.
Role of EMR	Three subcodes, as outlined below.	Maps to O3 interview questions 9 (what role can the EMR play in helping to improve process measures for diabetes care management?) and 10 (what role can the EMR play in helping to improve clinical indicators for diabetes care management?)
	Improve clinical indicators for DM	Maps to O3 interview question 10: what role can the EMR play in helping to improve clinical indicators for diabetes care management?
	Improve process measures for DM	Maps to O3 interview question 9: what role can the EMR play in helping to improve clinical indicators for diabetes care management?
	Improving patient care	The role that the EMR can play in improving patient care.
Usability	N/A	Comments related to improving usability of the EMR.

Abbreviations: DCM, designated contract market; EMR, electronic medical record; MU, meaningful use; N/A, not applicable; PCP, primary care physician.

Results

In total, 14 PCPs from the overall study participated in interviews (78%), and their participant demographics are outlined in [Table 6] below. Specifically, 12 PCPs participated in interviews at O3, and nine participated in interviews at O4. Of those PCPs who participated at O4, the majority (n = 7, 78%) had participated in the O3 interviews. The thematic analysis of the qualitative data revealed seven themes, which fall into two main categories: (1) design and development of EMR video tutorials, and (2) adoption and use of EMR video tutorials. A summary table of the themes can also be seen in [Table 7].

Table 6 Participant demographics

Characteristic	Number of participants (n = 14)	Percentage of participants (n %)
Age group
35–44	1	7.1
45–54	6	42.9
55–64	6	42.9
65+	1	7.1
Sex
Male	4	28.6
Female	10	71.4
Health authority/location
Region A	4	28.6
Region B	1	7.1
Region C	9	64.3
Years of practice
< 4	2	14.3
5–9	0	0.0
10–14	0	0.0
15–19	1	7.1
20+	11	78.6
Type of practice
Multiphysician	14	100
Solo	0	0
Years of EMR experience
1–2	1	7.1
3–4	5	35.7
5–9	8	57.1
10–14	0	0.0
Years of experience using OSCAR EMR
1–2	3	21.4
3–4	6	42.9
5–9	5	35.7
10–14	0	0.0
Number of EMRs used
1	5	35.7
2	6	42.9
3	2	14.3
4	1	7.1
Prior postimplementation training
Yes	7	50.0
No	7	50.0
Computer skills
Low	3	21.4
Average	8	57.1
Above average	3	21.4
High	0	0.0
EMR skills
Low	1	7.1
Average	10	71.4
Above average	3	21.4
High	0	0.0

Abbreviation: EMR, electronic medical record.

Table 7 Summary of thematic analysis themes

Theme	Main findings
PCPs liked the EMR video tutorials.	PCPs had a positive reaction to the video tutorials.
	PCPs liked the EMR video tutorial trainer. They found him to be a clear and knowledgeable expert who is good spokesperson for PCPs.
	PCPs did not feel anything was missing from the EMR video tutorials for diabetes care management.
	PCPs would recommend other PCPs to watch the EMR video tutorials.
PCPs would like more EMR video tutorials.	PCPs would like short, online EMR video tutorials on a breadth and depth of topics to support PCPs at different EMR use levels and in different communities.
	PCPs would like a searchable library of EMR video tutorials. It would be helpful for the video tutorials to be embedded in the EMR to increase accessibility at the point of care.
	PCPs think it may be helpful to develop EMR video tutorials for MOAs.
PCPs think it is important to have a PCP involved in the design and development of EMR video tutorials.	PCPs prefer to have a practicing PCP involved in the design of EMR video tutorials to develop short and simple learning content that is appropriate for the audience.
	PCPs would prefer a well-spoken, professional, engaging, and credible physician trainer who is an experienced EMR user that they can trust and relate to. Including the biography of the trainer with the EMR video tutorials may be helpful.
PCPs have visual preferences for the production and presentation of the video tutorials.	PCPs would like to continue seeing the on-screen EMR navigation in the video tutorials.
	PCPs would find it valuable to see the trainer in the EMR video tutorials (i.e., see video of trainer's face and not just their audio/voice) to increase the audience's connection with the trainer.
	PCPs may find it helpful to have written instructions available below the video tutorials.
PCPs suggest a multifaceted approach to promote the adoption of video tutorials for EMR optimization.	PCPs think it would be helpful for physician support organizations such as the Divisions of Family Practice and Pathways to host and create awareness of the EMR video tutorials.
	Financial and continuing professional development incentives may help PCPs to adopt the EMR video tutorials.
	PCPs would like additional end-user support sources to supplement the EMR video tutorials (e.g., follow-up reminder, in-person training, coaching, an online forum, etc.).
PCPs have experienced numerous barriers to integrating the EMR video tutorial training into practice.	Lack of time is the biggest physician-level barrier to integrating the EMR video tutorial training into practice for PCPs.
	Some PCPs chose not to use the “recalls” or “ordering labs” features for all their diabetic patients, due to the individual needs of their patients.
	PCPs have experienced many staff-level, patient-level, EMR-level, and policy-related barriers to integrating the EMR video tutorials training into practice.
	Several months after watching the EMR video tutorials, it was difficult for PCPS to remember what the training had covered.
PCPs have experienced numerous facilitators to integrating the EMR video tutorial training into practice.	Using the diabetes flow sheet and chronic disease management incentive fees are the biggest physician-level facilitators to integrating the EMR video tutorial training into practice for PCPs.
	PCPs have experienced many staff-level, patient-level, EMR-level, and policy-related facilitators to integrating the EMR video tutorial training into practice.

Abbreviations: DCM, designated contract market; EMR, electronic medical record; PCP, primary care physician.

PCPs View of the Video Tutorials

All participants had a positive reaction to the EMR video tutorials. For example, according to one participant: “so what I watched was very impressive” (participant 9). Specifically, they liked that the EMR video tutorials were “short,” “simple,” and “accessible.” Hosting the EMR video tutorials on YouTube also increased the accessibility of the video tutorials. PCPs found it to be a convenient way to access EMR training according to their schedules, as they “could do (the training) on (their) own time” (participant 6).

Several PCPs also found the video tutorials to be “helpful” and supporting effective visual learning by showing “on-screen” actions “because it is nice to see them go into the EMR and click on what we need to click on” (participant 13). Some PCPs also found the video tutorials to be a good reference or review because of the ability “to go back to (the EMR video tutorials)” (participant 4), especially when “you forget something you do not use very often, if you forget” (participant 4).

Many PCPs liked the EMR trainer. They found him to be a “clear” communicator. According to one PCP, the EMR trainer “was very enthusiastic and very clear and supportive. I thought he was an excellent person. He spoke slowly. He was fantastic. He showed things I thought it was very effective” (participant 3). Participants also considered the EMR video tutorial trainer to be a knowledgeable “expert of experts” who is a “good spokesperson” for PCPs.

In terms of the design of the video tutorials, PCPs liked the use of cuing to help with EMR interface navigation (e.g., highlighting of the chart) and would like to continue seeing it in EMR video tutorials because it is “very useful” (participant 15) and “helps you to focus your eyes where you are going...and circling the things you are supposed to go to” (participant 11).

All participants were pleased with the content of the EMR video tutorials, and did not think anything was missing for diabetes care, as it was “thorough,” “well done,” and “hit on everything (PCPs) would need for diabetes management” (participant 6). Participants universally stated that they would recommend other PCPs who use OSCAR EMR “to watch the (EMR video tutorials)” (participant 3).

Barriers and Facilitators

Barriers to Applying the EMR Video Tutorial Training to Practice

For many participants, lack of time was the biggest barrier to integrating the EMR video training into their practice. According to one participant using the EMR features (e.g., reminders) for diabetes care “takes time” (participant 13). Consequently, PCPs' use of the diabetes care features varied depending on their time and workload. Given the individual needs of diabetic patients, some PCPs did not use the “recalls” or “ordering lab” features for all of their diabetic patients “because each diabetic has something a little bit different that (the PCP wants) to order a little bit different lab work” (participant 2).

Participants have experienced staff-level barriers to applying the EMR video tutorials in practice, such as “computer literacy” and “having everybody in the office trained on the same system” (participant 1). At the patient level, the patient's responsibility in diabetes care was reported as a barrier because “there is a group of people that just are not going to be responsible” (participant 4). PCPs also experienced barriers at the EMR level, such as the EMR reminders being a “pull message” and not a “push message.” PCPs reported general EMR issues beyond training, as well. For example, for one PCP, the EMR was “not being as user-friendly as (the participant) would like” due to there being too many clicks (participant 9). From a policy perspective, there are barriers related to PCP funding (i.e., protected time) for EMR use because “(PCPs) are not remunerated for a whole lot of work (they) do in the job. Paper work, labs, referrals, EMR, on and on…there is not funding for that whatsoever. So it is a big barrier” (participant 9).

Facilitators to Applying the EMR Video Tutorial Training to Practice

For PCPs, the diabetes flow sheet was a key facilitator to integrating the EMR video tutorial training into practice because “it is so much easier than going through (PCPs') reams of paper...it is just right there” (participant 5). CDM incentive fees and a “desire to do good care” were also key facilitators “to adopt the (EMR video tutorials) fairly aggressively” (participant 12). PCPs' staff has supported applying the EMR video tutorial training in multiple ways, such as having “a (medical office assistant [MOA]) that is dedicated to doing the vitals and show people to their rooms” (participant 1). At the patient level, patients play a facilitating role in diabetes recalls by acting as a “safety net” so that between the PCP and the patient, “somehow (diabetes follow-up care) gets done” (participant 1). One participant shared that policy-related initiatives such as physician feedback serve as facilitators as well.

Multifaceted Approach to Promote the Adoption of EMR Video Tutorials

All PCPs suggested that the Divisions of Family Practice and Pathways (i.e., an online, searchable resource for GPs in BC who are members of a Division of Family Practice) would be best suited to (1) host the EMR video tutorials on their websites, and (2) create awareness about the video tutorials. A few participants suggested that financial incentives may facilitate adoption of the EMR video tutorials because “in the absence of (financial incentives), there is very little incentive to go and look at (EMR video tutorials) in people's spare time” (participant 7). Continuing Professional Development (CPD) credits were also suggested as a potential incentive for PCPs to be “more likely to do (the EMR video tutorial training)” (participant 5). One participant suggested that watching the EMR video tutorials should be a prerequisite for PCPs to receive incentive billing (e.g., for CDM).

To supplement the EMR video tutorials, PCPs would like to have access to additional end-user support sources and follow-up. For example, some participants would appreciate receiving follow-up reminders through the EMR or email to say “hey! look at this video hey! remember this training you did, and was there anything you needed to implement that you have not done yet? Do it. Contact us if you need help or something” (participant 6). Other participants would like to have access to in-person training “because (EMR) training always stimulates more questions, which are not automatically answered. So, in-person would be great. For like an hour or something” (participant 9). Individualized coaching was also suggested as form of end-user support; as well as online forums “as a place where you can go... if you do not have a person (to go to for support)” to ask EMR-related questions (participant 11).

Suggestions for Design and Development

PCPs Would Like More EMR Video Tutorials

Participants reported that they would like to receive more EMR video tutorials to support their increased use and optimization of the EMR and they would like these video tutorials to be short and similar to MD-PET: “… 2 to 3 minutes, and even shorter if possible so (PCPs) can figure out how to do something step by step…and have it organized in a way that could be searchable as well” (participant 2). Participants would also like the video tutorials to be publically available online.

Although PCPs appreciated MD-PET, they wanted EMR video tutorials on additional topics, such as “other chronic diseases like COPD” (participant 5), “determining an active medication list” (participant 13), and developing “forms” and “your own smart labs” (participant 2). Several PCPs also emphasized the need for having more video tutorials available for PCPs who are at different EMR use levels:

“No, I think they are good. You just need more. You need a whole batch of them with an index that physicians can go to get what they are missing because all of the physicians are at different levels of their comfort with the EMR.” (participant 2).

To support PCPs in different communities, one participant also recommended that video tutorials be developed for different communities in BC because “each community may have a different way of doing things and makes the (EMR video tutorials) not useful to them.” (participant 11).

Participants preferred the EMR video tutorials to be available as a searchable library that is “organized and...searchable so that somebody can just say “run reports” or “run diabetes report” and it would flick right to there” (participant 2) because “physicians use stuff that saves time but they will not use (the EMR video tutorials) if it is more time consuming” (participant 2). PCPs would also find it “even more helpful” if the library could be embedded within the EMR to increase accessibility at the point of care. For example, PCPs could “go to the help button and it (would have) all the list of the tutorials” (participant 13).

To support their MOA with increasing EMR use, PCPs suggested developing EMR video tutorials for MOAs because they would be “very helpful” and “MOAs would really appreciate it” (participant 3).

PCP Involvement in the Design and Development of EMR Video Tutorials

To ensure credible and trustworthy EMR training, participants highlighted the importance of having a PCP involved in the design and development of EMR video tutorials. They specifically preferred to have a practicing PCP who uses OSCAR EMR, is an “EMR specialist,” and “is aware of what the EMR can and cannot do” (participant 5).

Participants highlighted the importance of having a physician trainer who is well spoken, can “communicate effectively,” uses appropriate humor, is “concise,” and “to the point.” In terms of communicating, the physician trainer should be “speaking softly” and “enunciating their words.” Further, the physician trainer should be engaging, as “it's got to be lively, entertaining, well presented, clear, short” (participant 7). For participants, having a physician trainer “always lends credibility,” increases PCPs' ability to relate to the trainer and trust the EMR training, because “they understand, you know, [PCPs'] time constraints, and sort of the busyness of doing a practice” (participant 5). One participant described in detail this expected homophily with the physician trainer:

“So the more information that I find out that the person has in common with my situation, the more likely I am going to think they have something to offer me. So if I'm a community physician, you tell me this person is a community physician, and then I will probably listen to him more than if you told me somebody was a university lecturer. Because I'm thinking, how often do they use the system? Do they run into the same problems I do? And how much are they speaking to my level of function. So you have to target it to your audience.” (participant 11)

Additionally, to increase their “connection” and ability to relate with the trainer, several PCPs indicated that they would like to see video of the trainer's face and not just their audio (i.e., voice) because “[PCPs] seem to relate better to faces than to words. So it's nice having the person's face in the corner [of the EMR video tutorial screen]” (participant 15).

In addition to having knowledge, homophily, counseling, and communication skills, the PCP trainer should know “how to do things quickly and efficiently or offset the work to [Medical Office Assistant]” (participant 9). The trainer should also be “somebody with a very simple message. Very practical, demonstrating the skills...and not talking too much, just very direct and about the message you want to, the skill you want to get across” (participant 5).

Finally, several participants it would be valuable to know “the background of trainer, which physician in which location” and because it “would probably be interesting for the person learning” (participant 15).

Supplementary Written Instructions

As a learning supplement and reference, two PCPs stated that they may find it helpful to have written instructions (e.g., step-by-step instruction about how to record a blood pressure in the EMR) available below the EMR video tutorials. This would allow PCPs to watch the video tutorial or to read the written instructions “like a recipe” (participant 2). As such, the viewer “can watch the cook make it or [they] can read the instructions below” (participant 2).

Discussion

This study investigated the design and qualitative evaluation of postimplementation EMR video tutorials for diabetes care. Although video tutorials have recently been applied in a few studies to support EHR training for medical students, medical residents, and nursing students,[20] [21] [22] the design and use of video tutorials to support PCPs or physicians in general has not been widely investigated to date. Further, EMR video tutorials that are informed by the CCM, and are based on clinician-led training and best practices of software video tutorials' design have rarely been examined.

The present study revealed PCPs' unanimous liking of the video tutorials and subsequent desire to have access to more EMR video tutorials. Multiple barriers and facilitators to applying the EMR video tutorial training to practice were also found at the physician, staff, patient, EMR, and policy levels. However, none of the identified barriers specifically related to the design of the intervention, although some enhancements were suggested (e.g., use of a handout in conjunction with a video tutorial). Almost all of the identified barriers and facilitators align with the “barriers to and incentives for change at different levels of healthcare” of Grol and Wensing's framework,[47] including the innovation, individual professional, patient, social context, organizational context, and economic and political context. Lack of time was the biggest barrier to applying the EMR video tutorials to practice. This aligns with previous literature indicating that PCPs require more time for recommended CDM than is available for patients overall,[48] as well as time as a barrier to adopting EHRs.[49] [50] The barrier of adapting the practices demonstrated in video tutorials to individual patient needs maps to the “patient” level barrier of Grol and Wensing's framework. It also aligns with research on the need for individualized diabetes management instead of a “one-size-fits-all” approach and consideration for comorbidities (e.g., patients with diabetes and chronic kidney disease).[51] [52]

To support the future design and development of EMR video tutorials, PCPs would like to have a PCP involved in the process. This preference for clinician-led training reinforces the concept of homophily as a characteristic of end-user support,[38] as well as learning through modeling in social learning theory. The study results also align with Eyal and Rubin's[41] three essentials for learning discussed earlier: (1) homophily: PCPs' preference for having a physician trainer, (2) identification: participants' recommendation to provide biography of the physician trainer with the video tutorial, and (3) parasocial interaction: participants liked seeing the face (video footage) of the physician trainer in the EMR video tutorial.

Additionally, PCPs emphasized the continued need to see the EMR on-screen navigation in the EMR video tutorials. This visual preference may correspond with dual coding theory, which suggests that visual images, coupled with verbal information, aids learning.[30]

Study Limitations

The study employed a small sample that may not be generalizable to the whole population. However, the study provides rich insight into the design and evaluation of EMR video tutorials for PCPs. The present study did not examine effectiveness and additional research is needed to ascertain the design and effectiveness of video tutorials for improving EMR use. Additionally, the video tutorial intervention should fully apply all eight best practice guidelines for designing video tutorials.[37]

Conclusion

This study revealed that PCPs liked the EMR video tutorials for diabetes care and many of the design best practices implemented in them, and that they would like more EMR video tutorials on various topics and EMR use levels. This study offers a roadmap for health informatics professionals and instructional designers to develop EMR training videos for physicians and other EMR users with content and theory that meets evidence-based design criteria. The study results also help to identify opportunities to improve the design and delivery of EMR video tutorials for future training interventions including the development of an EMR video tutorial library, continued use of physician trainers in the design and development of EMR video tutorials, use of a multifaceted EMR training strategy, and provision of incentives to support adoption of video tutorials by PCPs. For future development and deployment of video tutorials, PCPs have suggested a multifaceted approach that includes additional end-user support to promote the adoption of EMR video tutorials. Policy makers and EMR vendors should consider investing in the development of EMR video tutorials informed by theory- and evidence-based best practices for designing software video tutorials. EMR users could benefit from developing an EMR video tutorial library as an open educational resource[53] for all EMRs, which can be hosted on an accessible website such as YouTube. Potential funding or CPDaccreditation for PCPs to watch the EMR video tutorials during protected time should be considered as well. Physician quality improvement programs may consider including EMR video tutorials as a supplementary end-user support resource. The development of EMR video tutorials for MOAs, as suggested by study participants, should also be considered.

Clinical Relevance Statement

Postimplementation EMR training for PCPs is generally inadequate and ineffective at present; to improve value-adding EMR use such as advanced features for diabetes care, training methods must be reliable, valid, feasible, educationally effective, and acceptable to users.[54] This study has demonstrated PCPs' endorsement and desire for video tutorials and video tutorial design best practices as a medium for providing advanced EMR training. They may also use the EMR video tutorials to train their office staff, as well as medical residents and locum physicians.

• References

• 1 Canada Health Infoway. The emerging benefits of electronic medical record use in community-based care; 1–98. Available at: https://www.pwc.com/ca/en/healthcare/publications/pwc-electronic-medical-record-use-community-based-care-report-2013-06-en.pdf . Accessed April 21, 2013
• 2 Survey NP. What electronic functions are you planning to use in the next two years? NR. 2014:2012. Available at: http://nationalphysiciansurvey.ca/wp-content/uploads/2014/09/2014-FPGP-EN-Q7iii.pdf . Accessed May 7, 2020
• 3 Commonwealth Fund. 2015 International Survey of Primary Care Doctors. 2015. Available at: https://international.commonwealthfund.org/data/2015/ . Accessed May 7, 2020
• 4 Denomme LB, Terry AL, Brown JB, Thind A, Stewart M. Primary health care teams’ experience of electronic medical record use after adoption. Fam Med 2011;43(09):638–642
• 5 Loomis GA, Ries JS, Saywell RM Jr, Thakker NR. If electronic medical records are so great, why aren’t family physicians using them? J Fam Pract 2002;51(07):636–641
• 6 Price M, Singer A, Kim J. Adopting electronic medical records: are they just electronic paper records? Can Fam Physician 2013;59(07):e322-e329
• 7 Randhawa GK, Lau F, Price M. Evaluating the adoption of e-prescribing in primary care. Healthc Q 2013;16(04):55–60
• 8 Swerissen H, Duckett S, Wright J. Chronic failure in primary medical care. 2016. Available at: https://grattan.edu.au/wp-content/uploads/2016/03/936-chronic-failure-in-primary-care.pdf . Accessed March 2, 2016
• 9 Canadian Institute for Health Information. Diabetes care gaps and disparities in Canada. Community Health (Bristol) 2009. Available at: http://secure.cihi.ca/cihiweb/products/Diabetes_care_gaps_disparities_aib_e.pdf . Accessed May 7, 2020
• 10 Edwards G, Kitzmiller RR, Breckenridge-Sproat S. Innovative health information technology training. CIN Comput Informatics Nurs 2012;30(02):104–109
• 11 Dastagir MT, Chin HL, McNamara M, Poteraj K, Battaglini S, Alstot L. Advanced proficiency EHR training: effect on physicians’ EHR efficiency, EHR satisfaction and job satisfaction. AMIA Annu Symp Proc 2012;2012:136–143
• 12 Bredfeldt CE, Awad EB, Joseph K, Snyder MH. Training providers: beyond the basics of electronic health records. BMC Health Serv Res 2013;13:503
• 13 Venkatesh V. Creation of favorable user perceptions: exploring the role of intrinsic motivation. Viswanath Venkatesh Source MIS Q. 1999;23(02):239–260
• 14 Nelson RR, Cheney PH. Training end users - an exploratory-study. MISQ Q. 1987;11(04):547–559
• 15 Baecker R. Showing instead of telling. In: ACM SIGDOC 2002. 2002:10-16. Available at: https://www.dgp.toronto.edu/papers/rbaecker_SIGDOC2002.pdf . Accessed May 7, 2020
• 16 Palaigeorgiou G, Despotakis T. Known and unknown weaknesses in software animated demonstrations (screencasts): a study in self-paced learning settings. J Inf Technol Educ 2010;9:1–18
• 17 Harrison S. A comparison of still, animated, or nonillustrated on-line help with written or spoken instructions in a graphical user interface. Proc SIGCHI Conf Hum Factor Comput Syst. Vol Denver, Colorado, USA: ACM Press/Addison-Wesley Publishing Co.; 1995:82-89. Available at: https://dl.acm.org/doi/fullHtml/10.1145/223904.223915 . Accessed May 7, 2020
• 18 Palmiter S, Elkerton J. An evaluation of animated demonstrations of learning computer-based tasks. Proc SIGCHI Conf Hum Factors Comput Syst. 1991:257-263. 10.1145/108844.108906. Available at: https://dl.acm.org/doi/10.1145/108844.108906 . Accessed May 7, 2020
• 19 Palmiter S, Elkerton J. Animated demonstrations for learning procural computer-based tasks. Hum Comput Interact 1993;8(03):193–216
• 20 He Z, Marquard J, Henneman E. Model guided design and development process for an electronic health record training program. AMIA Annu Symp Proc 2017;2016:1814–1821
• 21 Thiyagarajan A, Allen C, Peacock J, Cousins R. Implementing training videos for student clinicians to improve charting and utilization of EHR capabilities. 2017;3. Available at: http://www.themspress.org/journal/index.php/freeclinic/article/view/237 . Accessed May 7, 2020
• 22 Zoghbi V, Caskey RC, Dumon KR, et al. “How to” videos improve residents performance of essential perioperative electronic medical records and clinical tasks. J Surg Educ 2018;75(02):489–496
• 23 Randhawa GK, Shachak A, Courtney KL, Kushniruk A. Evaluating a post-implementation electronic medical record training intervention for diabetes management in primary care. BMJ Health Care Inform 2019;26(01):e100086
• 24 Bethke FJ, Dean WM, Kaiser PH, Ort E. Improving the usability of programming publications. IBM Syst J 1981;20(03). Doi: 10.1145/965660.965668
• 25 Van der Meij H, Carroll J. Principles and Heuristics for Designing Minimalist Instruction. Cambridge, MA: MIT Press; 1995
• 26 Horn R. Structured writing at twenty-five. Perform Instr 1993;32:11–17
• 27 Mayer R, Mayer RE. The Cambridge Handbook of Multimedia Learning. Cambridge, MA: Cambridge University Press; 2005
• 28 Tversky B, Morrison JB, Betrancourt M. Animation?: can it facilitate? Int J Hum Comput Stud 2002;247–262
• 29 Neumann O. Theories of attention. In: Neumann O, Sanders AF, eds. Handbook of Perception and Action. Vol. 3. Cambridge, MA: Academic Press; 1996:389–446
• 30 Paivio A. Imagery and Verbal Processes. NY: Holt, Rinehart, and Winston; 1971
• 31 Bandura A. Social Learning Theory. Englewood Cliffs, NJ: Prentice Hall; 1977
• 32 Bandura A. Social Foundations of Thought and Action: A Social Cognitive Theory. Upper Saddle River, NY: Prentice-Hall, Inc; 1986
• 33 van der Meij H, Gellevij M. The four components of a procedure. IEEE Trans Prof Commun 2004;47(01):5–14
• 34 Plaisant C, Shneiderman B. Show me Guidelines for producing recorded demonstrations. 2005. Available at: http://www.cs.umd.edu/hcil/trs/2005-02/2005-02.pdf . Accessed May 7, 2020
• 35 Brar J, Van Der Meij H. Computers in human behavior complex software training: harnessing and optimizing video instruction. Comput Human Behav 2017;70:475–485
• 36 Swarts J. New modes of help: best practices for instructional video. Tech Commun (Washington) 2012;59(03):195–206
• 37 van der Meij H, van der Meij J. Eight Guidelines for the design of instructional videos for software training. Tech Commun (Washington) 2013;60(03):205–228
• 38 Shachak A, Barnsley J, Tu K, Jadad AR, Lemieux-Charles L. Understanding end-user support for health information technology: a theoretical framework. Inform Prim Care 2011;19(03):169–172
• 39 Grol R, Wensing M. Effective implementation of change in healthcare: a systematic approach. 2013:40–59
• 40 Torre DM, Daley BJ, Sebastian JL, Elnicki DM. Overview of current learning theories for medical educators. Am J Med 2006;119(10):903–907
• 41 Eyal K, Rubin AM, Eyal K, Rubin AM. Viewer aggression and homophily, identification, and parasocial relationships with television characters viewer aggression and homophily, with television characters. J Broadcast Electron Media 2010. Doi: 10.1207/s15506878jobem4701
• 42 Randhawa G. OSCAR EMR study recruitment video. 2017. Available at: https://www.youtube.com/watch?v=zNQTWEyNU2c&feature=youtu.be . Accessed August 9, 2019
• 43 Singer A, Ivers NM. Using your electronic medical record to deliver evidence-based diabetes care. Can Fam Physician 2019;65(01):43–44
• 44 PAHO. The chronic care model. 2013. Available at: https://www.paho.org/hq/index.php?option=com_content&view=article&id=8502:2013-the-chronic-care-model&Itemid=1353&lang=en . Accessed April 4, 2016
• 45 Randhawa G, Yap J. OSCAR EMR video tutorial series. 2017. Available at: https://www.youtube.com/playlist?list=PLqNVRTDyA_99lsQzVXz4K3X8VvZr7oUXZ . Accessed September 27, 2017
• 46 QSR International Pty Ltd. NVivo qualitative data analysis software, version 10. 2012. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3930250/ . Accessed February 12, 2014
• 47 Grol R, Wensing M. What drives change? Barriers to and incentives for achieving evidence-based practice. Med J Aust 2004;180(S6):S57–S60
• 48 Østbye T, Yarnall KSH, Krause KM, Pollak KI, Gradison M, Michener JL. Is there time for management of patients with chronic diseases in primary care? Ann Fam Med 2005;3(03):209–214
• 49 Ajami S, Bagheri-Tadi T. Barriers for adopting electronic health records (EHRs) by physicians. Acta Inform Med 2013;21(02):129–134
• 50 Shuval K, Berkovits E, Netzer D, et al. Evaluating the impact of an evidence-based medicine educational intervention on primary care doctors' attitudes, knowledge and clinical behaviour: a controlled trial and before and after study. J Eval Clin Pract 2007;13(04):581–598
• 51 Bailey CJ, Aschner P, Del Prato S, LaSalle J, Ji L, Matthaei S; Global Partnership for Effective Diabetes Management. Individualized glycaemic targets and pharmacotherapy in type 2 diabetes. Diab Vasc Dis Res 2013;10(05):397–409
• 52 Levey AS, Coresh J. Chronic kidney disease. Lancet 2012;379(9811):165–180
• 53 United Nations Educational Scientific and Cultural Organization. What are open educational resources? 2017. Available at: https://en.unesco.org/themes/building-knowledge-societies/oer . Accessed May 7, 2020
• 54 Shachak A, Reis S, Gray K, et al. Evaluating educational interventions for health professions in the digital age. In: Shachak A, Borycki EM, Reis SP, eds. Health Professionals’ Education in the Age of Clinical Information Systems, Mobile Computing and Social Networks. Amsterdam, Netherlands: Elsevier Inc.; 2017:471–488