Serious Games in Health Professions Education: Review of Trends and Learning Efficacy

Keywords

Education, medical - health occupations - learning - games - outcome assessment

Introduction

The 21^st century saw the emergence and implementation of digital education in health professions education. With demonstrated impact on patient safety [1] through medical simulations, and case studies describing improved learning outcomes, it is therefore unsurprising that there is an increased research effort into digital learning and emerging technologies in health professions education. Serious games (SG) area is an area that has received increased attention. According to Bergeron [2], a serious game is an “interactive computer application, with or without significant hardware components,” that is challenging, engaging, and that supplies the user with competencies useful in reality. To date, there is an expanding use of serious games in professional training, ranging from laparoscopy [3] to professional behaviors and attitudes [4]. The practice of medicine is one that requires precision and consistency to provide effective treatment while ensuring patient safety. Health professions education should, therefore, be held to strict and reproducible standards.

Despite the increased adoption of serious games in health professions education, current literature still mostly includes independent case studies only. Most review articles and proceedings publications about serious games in health professions education raise questions on their pedagogical methods, assess the evidence of their educational effectiveness, or examine how they are employed in health professions education [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16]. Most articles examine serious games targeted on health promotion for the general population and patients. Few articles examine serious games used exclusively by under-, postgraduate medical students, medical doctors, and allied health [5] [7] [8] [11] [16]. One review article examined video games and publication trends of the medical literature, but it focused on the possible biases of the researcher’s attitudes toward video games [17]. We did not find a review article that examines the trends of the use of serious games for the education of health professionals, using publication metadata. Furthermore, reviews on the efficacy of serious games for health professions education is limited [8], and their effects are primarily extrapolated from publications that described serious games in a much broader context [17].

This paper aims to provide an overview of the research trends and a scoping review of the efficacy of serious games for health professions education with regards to short-term learning outcomes. For a more comprehensive review, we will also be answering the two following research questions: (i) What are the main publication trends for SG? and (ii) What is the reported learning efficacy of SG with regards to the type of competency trained and the different health professions?

Methods

Review of Serious Games Publication Trends

We performed a search through the Web of Science core collection database and decided to limit the publication date from 1996 as most earlier publications did not have abstracts available. We examined the metadata on serious games in health professions education and narrowed it to the education for medical doctors as undergraduate, postgraduate, and medical specialties, dental, pharmacy, nurse education, and allied health. The reason for that definition was that the scope of knowledge in these professions has overlapping areas and skills. Even though there are reviews on the topic, we did not find any that examined the trends in the field for the two past decades using metadata. One of the key insights is that the domain of serious games is not an established field of study and therefore often not used as a term or a concept. Therefore, we decided to use the broader term of “games” as well as derivatives such as “gamification” for the search. We included publications about “digital games” and only the simulations which had gaming elements and excluded the ones about virtual reality and e-learning applications [19]. While both games and health professions education have many publications, when you combine the two it leads to a small subset of papers. About 1.7% of game-related publications are related to health professions education, and about 0.04% of health professions education publications are related to games.

Scoping Review of Serious Games Learning Efficacy

A scoping review was performed on current literature regarding serious games and health professions education. Health professions education is the formal training of personnel involved in patient care – from undergraduates to full-time doctors and nurses. Short-term learning refers to the performance of learners within the confines of the game and immediately after. The ability to achieve target outcomes define efficacy. In this context, serious games can produce improved short-term learning. This will be assessed through learners’ scores. Given the nature of the review, no ethical consideration was required.

To triangulate the field, we sourced literature from ERIC for educational psychology, Web of Science for top journals, and Medline for a clinical perspective. Other used databases included Education Source, PsychINFO, Global Health, and CINAHL. Key-terms (“serious gam*”) AND (educat* OR train* OR teach*) AND (medical OR medicine OR surge* OR surgical OR physician OR healthcare OR doctor* OR nurs* OR “allied health”) were searched by topic. The search was last performed on 30^th August 2018. Reviews, interviews, editorials, letters, and meeting abstracts were excluded from the search. Non-English language articles were also excluded. Dentistry and professions without international regulation (i.e., allied health) were excluded as well.

After an initial screening via titles and abstracts, the remaining studies were evaluated in full-text and classified based on competency and target group. The format to categories “competency” is derived from Rice and Sinclair [20] who described competency as “Knowledge, Skills, and Attitudes”. This method of categorization has seen widespread use in other educational research including Kirkpatrick’s four levels [21] and Miller’s prism of clinical competence [22]. In this paper, we further characterized “Skills” as either “Technical Skills” (the manual dexterity when performing procedures) or “Cognitive Skills” (the ability to correctly apply knowledge to achieve desired outcomes, e.g., diagnostic reasoning). “Knowledge” is defined as content knowledge on a particular topic, whereas “Attitudes” refers to the affective domain comprising personal beliefs and behaviors including ethics, patient safety, and interpersonal skills. The target group was categorized in two ways: by health professions (medicine or nursing), and by the level of education (undergraduates or postgraduates).

Results

Serious Games Publication Trends

We attempted to identify the very early efforts, the origins of the first research in serious games in health professions education. As we explained, the term “serious games” is not well established and therefore a lot of publications, especially earlier ones, do not explicitly refer to their research as serious games, but as video games. The first paper in the dataset we examined is from 1996 and discusses a simulation game in pre-registration nurse education [23]. The first paper that explicitly calls out “serious games” in health professions education in its title comes 11 years later in a paper by Case Western Reserve University [24].

We noticed an overall increase in literature from 1996 to 2018, which demonstrates a healthy growth in the interest of this subject. We can distinguish 3 phases: (i) the exploratory phase up to 2006, (ii) the early growth from 2007 to 2012, and (iii) the development phase from 2013 onwards ([Figure 1]).

Fig. 1 Publications trends are composed of three main phases: (i) the exploratory phase up to 2006, (ii) the early growth from 2007 to 2012, and (iii) the development phase from 2013 onwards.

From 1996 to 2006 (11 years), 14 articles were published. In the following period, from 2007 to 2012 (6 years), 77 articles were published, and in the last and recent period from 2013-2017 (5 years) 159 articles were published. So, we have moved from on average of 1.3 articles/year during the first period, to 12.8 articles/year during the second period, and 31.8 articles/year in the more recent period. Even though the articles published decreased in 2017, it seems that it picked up again in 2018.

With regards to the key conferences targeted by publications on this topic, we can see a diversity of conferences. However, one stands out, which produced the largest number of publications in our collection, the IEEE International Conference on Serious Games and Applications for Health which has been held on most years since 2011.

Most of the publications written on the topic come from the USA (n=91). When looking at the geographical area of North America, i.e., combining the USA and Canada, we have 122 articles. The most productive European country on the topic is the UK (n=30). When we add the articles by all European countries though, we see that they are 128, which makes Europe the one most active area in the topic. Asian countries have 13 publications in the group, led by China and Singapore.

Several articles spread over more than one research area, such as surgery and health care sciences. Of the medical specialties, the most researched topics are surgery (n=39) and nursing education (n=30), followed by general internal medicine and rehabilitation.

Serious Games Learning Efficacy

We got a total of 775 retrieved papers. The [Figure 2] displays the whole process of the literature search. Only 25 studies addressed short-term learning in serious games for health professions education. [Table 1] gives an overview of these publications. They spanned from 2010 to 2018 and described 23 different serious games. Twenty-two were quantitative analyses, and three were qualitative. Regarding experimental design, 15 were Randomized-Controlled Trials (RCTs), and the remaining ten were quasi-experimental. Of the 25 publications, 16 had both pre-test and post-test evaluations. Collation of empirical results showed that all these 16 studies demonstrated serious games for health professions education to be efficacious with regards to short-term learning outcomes. This is due to a significant increase in post-test scores over pre-test.

Fig. 2 Search strategy for literature on serious games (SG) for health professions education with regards to short-term learning (STL) outcomes.

Table 1 Overview of the 25 studies that address short-term learning outcomes in serious games for health professions education.

Paper	Aim of the paper	Serious Game	In-Game Objectives	Research Design	Competency	Target Group
1 [25]	To examine the effects of fidelity of open patient cases as an adjunct to an online course, on leaner's cognition and motivation.	abcdeSIM	The serious game challenges the players to perform the ABCDE approach on virtual patients and provides virtual tools for physical examination and investigations. The objective is to accurately interpret findings and manage the patient promptly (15 minutes).	RCTB	K + S (C)	Medical Students#
2 [26]	To assess cognition and motivation between the use of course manual versus an additional serious game in teaching emergency medicine.			Quasi-experimentalA	K + S (C)	Doctors&
3 [27]	Reporting of the preliminary assessment of InsuOnline©.	InsuOnline©	The player is challenged using an increasing complexity series of cases involving diabetes and is tasked to choose the best management option to improve diabetic control. The game provides immediate feedback after each action and allows progression only if sufficient correct decisions are made.	Quasi-experimentalA	K	Medical Students#; Doctors&
4 [28]	To compare the appropriateness, acceptability, and effectiveness of InsuOnline© against continual medical education on diabetic medicine education.			RCTA	K	Doctors&
5 [29]	To assess improvements in laparoscopy competence using the game “Underground”.	Underground	Players are tasked to free robots which are stuck underground and guide them to the surface while navigating through obstacles using game controllers that mimicked the movements of laparoscopic surgery.	RCTA	S (T)	Medical Students#
6 [3]	To compare the efficacy of using “Underground” versus control in teaching laparoscopy.			Quasi-experimentalB	S (T)	Doctors&
7 [30]	A report on the crafting and utilization of the UT TIME Portal, as well as the related experimental outcomes.	UT TIME	To improve interviewing skills, players have to interact with a virtual patient and ask for relevant symptoms, while choosing when to continue listening, check for accuracy, or transition to a different symptom attribute.	Quasi-experimentalC	A	Medical Students#
8 [31]	To contrast the educational outcomes between traditional Advanced Cardiac Life Support (ACLS) lessons versus ACLS simulations using virtual reality	ACLS virtual reality simulator	In this multiplayer game, each player takes on a role in the resuscitation team, with unique objectives (e.g., chest compression, airway). The team has to evaluate the patient clinically and resuscitate accordingly.	RCTA	K + S (C & T)	Doctors&
9 [32]	To assess the differences in outcomes with regards to cognition, attention to patient safety, and motivation between the users of “Air Medic Sky-1” versus an online course.	Air Medic Sky-1	The game teaches patient safety awareness through communication and stress management skills. Players have to treat multiple patients simultaneously while implementing breathing techniques to manage stress, and communicating effectively with the virtual healthcare team.	RCTB	A	Medical Students#
10 [33]	To analyze educational outcomes after the use of a mixture of laboratory and computer simulations regarding the clinical assessment of the newborn amongst nursing undergraduates.	eBaby SSRNPT	The game virtually simulates a preterm infant with respiratory problems maintained in the incubator. Participants are challenged to assess the preterm infant and intervene accordingly clinically.	Quasi-experimentalC	S (C)	Nursing Students#
11 [34]	To analyze the experimental outcomes of the “Critical Transport” serious game.	Critical Transport	Players have to clinically evaluate the patient and decide the right crew and equipment to transport the patient, using recommendations from the Portuguese Society of Intensive Care.	Quasi-experimentalC	K, S (C)	Medical Students#
12 [35] 13 [36]	To assess the understanding and proficiency in the use of Vancomycin amongst healthcare professionals. To report on the crafting, utilization, and experimental outcomes of eMedOffice.	Vancomycin interactive eMedOffice	Players have to make choices based on scenarios depicted in video clips of patient-doctor interactions. These choices aim at testing the knowledge regarding the use of Vancomycin in managing Methicillin-Resistant Staphylococcus Aureus (MRSA). In this 2D simulation, players have to optimize the interior design of a medical clinic. Virtual workers and patients feedback on the layout, allowing subsequent troubleshooting and improvement.	Quasi-experimentalC Quasi-experimentalC	K K	Pharmacists, Nurses&, Doctors& Medical students#
14 [37]	To evaluate the efficacy of “EMSave” serious game for ALS retraining.	EMSave	The player plays as the leader of an ALS team and has to clinically evaluate the patient and provide effective intervention. He also has to communicate effectively with the patient and bystanders for history taking.	Quasi-experimentalC	K + S (C)	Nurses&, Doctors&
15 [38]	To evaluate the use of “Touch Surgery” to educate medical students on thoracostomy.	Touch Surgery	Players are guided through surgical procedures via touchscreen motion gestures on the phone. Learning is assessed via rehearsal of the gestures.	RCTB	S (T)	Medical students#
16 [4]	To assess if serious games can enhance surgeons' reaction to apparatus or machinery failure during surgery.	Dr. Game, Surgeon Trouble	Players have to create rows of three similar blocks. Sporadically, changes in the game environment occur outside the players' direct focus of attention. Players have to recognize them and respond accordingly for bonus points.	RCTB	A	Doctors&
17 [39]	To assess the effect of serious games on intermediate life support (ILS) outcomes.	PULSE	Players are faced with choices that test understanding of equipment and intermediate life support principles through clinical scenarios. The timeliness of the response and actions translates to the player's score.	RCTB	K + S (C)	Nursing Students#
18 [40]	To assess “GeriatriX” in teaching effective clinical judgment regarding patient autonomy, applicability, and price of healthcare.	GeriatriX	Through three clinical scenarios depicting the same clinical issues but different patient preferences and socio-economic factors, players have to choose the optimal diagnostic and therapeutic strategy for each of them.	RCTA	K	Medical Students#
19 [41]	To compare the effectiveness of an e-module versus a serious game, on top of simulation, to teach the management of cardiac arrest.	Staying Alive	The player interacts with a virtual patient who has just experienced a cardiac arrest. Through this interaction, the player can take actions to learn appropriate behavior, movements, and techniques of cardiopulmonary resuscitation (CPR).	RCTA	S (T)	Medical Students#
20 [42]	To assess if “Orthotopic Liver Transplant (OLT) Trainer” will enhance surgeon performance and educate best practices.	OLT Trainer	The player has to bring patients through phases of surgery, from preoperative assessment to ICU admission. The player uses in-game funds to perform actions and to purchase equipment and tests to complete each phase.	RCTA	K + S (C)	Doctors&
21 [43]	To assess the comparative effectiveness of triage trainer and conventional training methods with regards to major incident. triage.	Triage Trainer LISSA	The player is the first responder to a bomb explosion scene and has to identify and clinically evaluate the casualties, as well as assign the appropriate priority to each of them.	RCTB RCTB	K + S (C)	Doctors&, Nurses& and paramedics
22 [44]	To illustrate and assess the effectiveness of LISSA in CPR education for nursing students.	LISSA	Player has to provide CPR to a virtual patient via keyboard inputs, and is assessed by timeliness, rhythm, and depth.	RCTB	S (T)	Nursing Students#
23 [45]	To compare using a V.R. system versus lecture-styled education in educating nursing students on pediatric respiratory content.	N.A.	Players have to accurately clinically evaluate virtual pediatric patients with respiratory problems and provide intervention. Players have to hand-off in the SBAR format at the end of the scenario.	RCTB	Phase 1: K Phase 2: K + S (C&T	Nursing Students#
24 [46]	To assess the effectiveness of a serious game to enhance knowledge of blood transfusion provision amongst undergraduate nurses.	N.A.	Player has to complete the pre-transfusion, transfusion, and post transfusion phases regarding administering a blood transfusion. This is done through mini-games that test his/her knowledge at each phase.	RCTA	K + S (C)	Nursing Students#
25 [47]	To evaluate the effectiveness of a serious game in improving learner confidence in dealing with post-partum hemorrhage.	N.A.	In this multiplayer game, player operates in teams to clinically evaluate (through investigations and physical examination) and manage virtual patients with post-partum hemorrhage.	Quasi-experimentalC	K + S (C & T) + A	Doctors&
Legend

RCT^A = Randomized-Controlled Trials with pre-test & post-test

RCT^B = Randomized-Controlled Trials with post-test without pre-test

Quasi-experimental^A = Quasi-experimental with control, pre-tests & post-test

Quasi-experimental^B = Quasi-experimental with control & post-test, without pre-tests

Quasi-experimental^C = Quasi-experimental with pre-test & post-test, without control

K = Knowledge

S (T) = Technical Skills

S (C) = Cognitive Skills

A = Attitudes

# = Undergraduates

& = Postgraduates

Eighteen of the 25 papers were controlled experiments using traditional methods of education. All but four of the 18 studies documented significantly higher post-test scores after serious games compared to the control. This suggests that serious games for health professions education were more efficacious than conventional teaching methods. The four dissenting publications (publications number 1, 8, 9, and 19 in [Table 1]) concluded that serious games were only as efficacious as their traditional counterparts. [Figure 3] summarizes the efficacy of SG based on competencies. [Table 2] describes the efficacy of serious games based on health professionals targeted.

Fig. 3 Classification of publications based on the competency addressed. The number in each field shows the study count for each class. Solid fields belong to one class. The field with top-left to bottom-right diagonal lines represents studies with classes Knowledge and Skills (cognitive). The field with top-right to bottom-left diagonal lines represents studies with classes Knowledge, Skills (cognitive) and Skills (technical). The crosshatched field show the publication belonging to all four classes. The number in brackets represents the number of articles that documented no significant difference in short-term learning between serious games and control. Example: there are 5 publications on serious games that only train the competency of Technical Skills, of which 1 describes that serious games are only as effective, but not significantly better than control at achieving short-term learning outcomes

Table 2 Classification of publications by health professionals. The number in brackets represents the number of articles that documented a significant difference in short-term learning between serious games and control.

Domain	Level	Number of studies
Medicine	Undergraduate	9 (7)
	Postgraduate	7 (6)
	Both	1(1)
Nursing	Undergraduate	5 (5)
	Postgraduate	0 (0)
	Both	0 (0)
Medicine & Nursing		3 (3)

Discussion

With regards to the publication trends, the gaming field could be regarded as still emerging, getting an accelerated interest by researchers. Serious games applied to patients, and the general population, have attracted a broader research interest than serious games for medical practitioners.

The United States dominate research but Europe, as a whole, has a predominant position. Asian researchers are not significantly represented, so there is an area of opportunity for this part of the world. There are no firmly established journals and conferences that focus on serious games in health professions education. Key areas serious games are applied to are surgery and nursing education.

The review of learning efficacy showed that serious games for health professions education are efficacious for short-term learning. The finding is consistent with studies by Sipiyaruk [48] and Connolly [49]. The former described the teaching of medicine and dentistry using serious games, and the latter was a review of the use and impact of computer and serious games in a broader context.

This study further suggests that serious games could even be more efficacious than conventional methods of health professions education. According to user-experience surveys in 11 of the 25 articles [25] [26] [27] [28] [32] [35] [36] [37] [39] [40] [46], many reported an increased sense of engagement and immersion. This was commonly credited to the competitive element, fun, and interactivity of the games. Undoubtedly, serious games when designed appropriately have the edge over conventional teaching methods regarding user experience and, in extension, player retention and learning. Furthermore, serious games also allow “stealth learning” [50] which is explained by Wouter as “when the trainee is too busy having fun to notice an improvement on key educational outcomes” [3]. These reasons could explain the significantly higher post-test scores after serious games. However, evidence supporting this claim is not consistent as four of the 18 controlled experiments stated otherwise. On the other hand, these inconsistencies could be attributed to limitations such as ceiling effect [25] and small sample sizes [31].

However, the favorable findings described above when taken at a general level calls for a more in-depth analysis based on target groups and competencies. We have undertaken that analysis which revealed many gaps to address to be conclusive about efficacy. These gaps will be discussed in the next two sections below.

Competencies Addressed by Serious Games

From [Figure 3], there is significant clustering and overlap of existing literature about “Knowledge,” “Technical skills” and “Cognitive skills” competencies. These overlaps could signify the transition of learning as described by Miller’s prism of clinical competence [22] from “knows” to “shows,” also represented by the thick arrow in [Figure 3]. An example is the serious game for Advanced Cardiac Life Support (ACLS) in paper 8 [31] ([Table 1]). Firstly, learners had to understand the pathologies and principles of ACLS (Knowledge). Next was the ability to reach a correct diagnosis through diagnostic reasoning (Cognitive skills). Last, learners had to familiarize themselves with the provision of life-saving interventions (Technical skills).

The transition of learning is described by Robert E. Haskell [51] as the “use of past learning when learning something new and the application of that learning to both similar and new situations”. It is also “universally accepted as the ultimate aim of teaching”. This repeated transfer and reinforcement of information from one competency to another would only serve to strengthen one’s understanding and retention. The transition of learning is, after all, “the key to all effective instruction and learning” [51]. In the same vein, the transfer of said learning to the affective domain should only serve to fortify one’s understanding further.

However, it is evident from [Figure 3] that there is a gap in current literature concerning the integration of “attitudes” with other competencies when designing serious games for health professions education. This is reflected by the minimal overlap between “Attitudes” and the three other competency types. Therefore, we hypothesize that serious games for health professions education, with the integration of learning in the affective domain, will achieve more efficacious short-term learning outcomes compared to without.

Variation among Health Professionals

[Table 2] classifies the publications based on target groups. There is an absence of research on serious games for postgraduate nursing education. This could perhaps be attributed to the lack of a rigorous and significantly demanded postgraduate nursing curriculum, unlike medicine which has residency programmes and CME (Continuing Medical Education) sessions. Serious games for postgraduate nursing education is, therefore, a potential gap for future research. Given the indispensable role of nurses in the healthcare team, keeping their competencies up-to-date through education is therefore essential.

[Table 2] shows that serious games are efficacious for the medicine target group, including both undergraduates and postgraduates. Independently, serious games also appear to be efficacious for undergraduate nursing. However, more high-quality literature for postgraduate nursing education is required before one can draw reliable conclusions on the efficacy of nursing education as a whole.

Limitations of the Study

One limitation of the scoping review is the narrowed scope of its methodology. For greater coherence across each profession, we only selected those that are internationally regulated. As a result, publications on allied health were excluded. Doing so has limited the scope of this review and rendered it potentially inapplicable to the entire healthcare sector.

Another limitation is the lack of high-quality literature. Ten of the 25 publications are quasi-experimental and hence ascribed a lower level of evidence (LOE) than the 15 RCTs. However, due to the small sample size of literature, this disparity in LOE was not accounted for in the results. Therefore, more high-quality research in this field is required for a more rigorous methodology.

Future Work

Apart from addressing the two gaps identified above, there is also value in evaluating game design to understand what exactly makes a game useful.

To date, research on developing a common framework to evaluate serious games is still limited. Conventional approaches such as Bloom’s Ordered Thinking Skills [52] and Kirkpatrick’s four Levels [21] are often used, but these models served to be non-specific when applied to serious games. In response, academics have proposed RETAIN [53] and MEEGA+ [54] to promote a new design paradigm for this platform. However, there is still an absence of a clear standard especially in the field of serious games for health professions education.

Therefore, future work could look into designing a framework specific to the evaluation of serious games for health professions education – in particular how game mechanics and learning mechanics tie in to achieve learning. To accomplish this, we could increase the granularity of the methodology to identify relevant mechanics in well-established serious games. This could potentially shift the focus of current research from the promotion of games to one that discusses game design, enabling earlier intervention and the development of more efficacious serious games in the future.

Conclusions

This paper supports the view that serious games can work for short-term learning in the domain of health professions education. Most of the literature reviewed in this paper has demonstrated that serious games can potentially be even more efficacious than conventional teaching methods. This may be attributed to the engaging and immersive nature of games, as well as “stealth learning”. However, despite the positive implications, it is still too soon to apply blanket statements of efficacy across all serious games for health professions education. Categorization of these games into competencies and target groups reveals many gaps that have yet to be bridged. Until then, it will be difficult to draw reliable and sweeping conclusions. The two gaps identified will also serve as potential areas for future research for a more comprehensive understanding in this field.

• References

• 1 Ziv A, Small SD, Root Wolpe P. Patient safety and simulation-based medical education. Med Teach 2000; Jan 1 22 (05) 489-95
• 2 Bergeron BP. Developing Serious Games. Hingham, Mass: Charles River Media. ; c2006. ISBN:1584504447
• 3 van Goor H, Luursema J-M, Ijgosse WM. Saving robots improves laparoscopic performance: transfer of skills from a serious game to a virtual reality simulator. Surg Endosc 2018; 32 (07) 3192-9
• 4 Graafland M, Bemelman W, Schijven M, Bemelman WA, Schijven MP. Game-based training improves the surgeon’s situational awareness in the operation room: a randomized controlled trial. Surg Endosc 2017; 31 (10) 4093-101
• 5 Blakely G, Skirton H, Cooper S, Allum P, Nelmes P. Educational gaming in the health sciences: systematic review. J Adv Nurs 2009; Feb; 65 (02) 259-69
• 6 Adams S. Use of “serious health games” in health care: a review. Stud Health Technol Inform 2010; 157: 160-6
• 7 de Wit-Zuurendonk LD, Oei SG. Serious gaming in women’s health care. BJOG 2011 Nov; 118 Suppl 3 17-21
• 8 Graafland M, Schraagen JM, Schijven MP. Systematic review of serious games for medical education and surgical skills training. Br J Surg 2012; Oct; 99 (10) 1322-30
• 9 Bellotti F, Kapralos B, Lee K, Moreno-Ger P, Berta R. Assessment in and of Serious Games: An Overview. Advances in Human-Computer Interactio, vol. 2013
• 10 Ricciardi F, DePaolis LT. A Comprehensive Review of Serious Games in Health Professions. International Journal of Computer Games Technology, vol. 2014
• 11 Abdulmajed H, Park YS, Tekian A. Assessment of educational games for health professions: a systematic review of trends and outcomes. Med Teach 2015; Apr; 37 Suppl 1 S27-32
• 12 Ahmadi S, Derakhshan HB, Yazdani S, Hosseini M. The use of Gamification in Medical Educations. International Journal of Pharmaceutical Research and Allied Sciences 2016; 5 (02) 355-62
• 13 McCoy L, Lewis JH, Dalton D. Gamification and Multimedia for Medical Education: A Landscape Review. J Am Osteopath Assoc 2016; Jan; 116 (01) 22-34
• 14 Gorbanev I, Agudelo-Londoño S, González RA, Cortes A, Pomares A, Delgadillo V. et al. A systematic review of serious games in medical education: quality of evidence and pedagogical strategy. Med Educ Online 2018; Dec; 23 (01) 1438718
• 15 Pront L, Müller A, Koschade A, Hutton A. Gaming in Nursing Education: A Literature Review. Nurs Educ Perspect 2018; Jan/Feb; 39 (01) 23-28
• 16 Wang R, DeMaria S Jr, Goldberg A, Katz D. A Systematic Review of Serious Games in Training Health Care Professionals. Simul Healthc 2016; Feb; 11 (01) 41-51
• 17 Sergev A, Rovner M, Appel DI, Abrams AW, Rotem M, Bloch Y. Possible Biases of Researchers’ Attitudes Toward Video Games: Publication Trends Analysis of the Medical Literature (1980-2013). J Med Internet Res 2016; Jul 18 18 (07) e196
• 18 Girard C, Ecalle J, Magnan A. Serious games as new educational tools: how effective are they? A meta-analysis of recent studies. J Comput Assist Learn 2013; Jun; 29 (03) 207-19
• 19 Bazakidi E, Zary N. Serious Games Research Publications Trends: Review Protocol 2019. available at: https://osf.io/cg86k/ ; accessed March 30^th
• 20 Rice CA, Sinclair M. Competency-based objectives for clinical training. Can J Med Technol 1995; 57 (03) 136-40
• 21 Kirkpatrick DL, Kirkpatrick JD. editor Evaluating training programs. San Francisco: TATA Mcgraw Hill. ix-3; 2006
• 22 Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990; Sep 1; 65 (09) S63-7
• 23 Wildman S, Reeves M. The utilization and evaluation of a simulation game in pre-registration nurse education. Nurse Educ Today 1996; 16 (05) 334-9
• 24 Nosek TM, Cohen M, Matthews A, Papp K, Wolf N, Wrenn G. et al. A Serious Gaming/Immersion Environment to Teach Clinical Cancer Genetics. Stud Health Technol Inform 2007; 125: 355-60
• 25 Dankbaar ME, Alsma J, Jansen EE, van Merrienboer JJ, van Saase JL, Schuit SC. An experimental study on the effects of a simulation game on students’ clinical cognitive skills and motivation. Adv Health Sci Educ 2016; 21 (03) 505-21
• 26 Dankbaar ME, Roozeboom MB, Oprins EA, Rutten F, van Merrienboer JJ, van Saase JL. et al. Preparing Residents Effectively in Emergency Skills Training With a Serious Game. Simul Healthc 2017; Feb; 12 (01) 9-16
• 27 Diehl LA, Gordan PA, Esteves RZ, Coelho IC. Effectiveness of a serious game for medical education on insulin therapy: a pilot study. Arch Endocrinol Metab 2015; Oct; 59 (05) 470-3
• 28 Diehl LA, Souza RM, Gordan PA, Esteves RZ, Coelho IC. InsuOnline, an Electronic Game for Medical Education on Insulin Therapy: A Randomized Controlled Trial With Primary Care Physicians. J Med Internet Res 2017; Mar 9 19 (03) e72
• 29 Harrington CM, Chaitanya V, Dicker P, Traynor O, Kavanagh DO. Playing to your skills: a randomised controlled trial evaluating a dedicated video game for minimally invasive surgery. Surg Endosc 2018; 32 (09) 3813-21
• 30 Zielke MA, Zakhidov D, Jacob D, Hardee G. Beyond Fun and Games: Toward an Adaptive and Emergent Learning Platform for Pre-Med Students With the UT TIME Portal. In: 2016 IEEE International Conference on Serious Games and Applications for Health (SeGAH). p.1-8
• 31 Khanal P, Vankipuram A, Ashby A, Vankipuram M, Gupta A, Drumm-Gurnee D. et al. Collaborative virtual reality based advanced cardiac life support training simulator using virtual reality principles. J Biomed Inform 2014; Oct; 51: 49-59
• 32 Dankbaar ME, Richters O, Kalkman CJ, Prins G, Ten Cate OT, van Merrienboer JJ. et al. Comparative effectiveness of a serious game and an e-module to support patient safety knowledge and awareness. BMC Med Educ 2017; Feb 2 17 (01) 30
• 33 Fonseca LM, Aredes ND, Fernandes AM, Batalha LM, Apóstolo JM, Martins JC. et al. Computer and laboratory simulation in the teaching of neonatal nursing: innovation and impact on learning. Rev Latino-Am Enfermagem 2016; 24: e2808
• 34 Ribeiro C, Antunes T, Pereira J, Monteiro M. Critical Transport: A Serious Game to Teach the Recommendations for the Transport of Critically Ill Patients. Gamification: Concepts, Methodologies, Tools, and Applications. IGI Global 2015; p. 1586-608
• 35 Bond SE, Crowther SP, Adhikari S, Chubaty AJ, Yu P, Borchard JP. et al. Design and Implementation of a Novel Web-Based E-Learning Tool for Education of Health Professionals on the Antibiotic Vancomycin. J Med Internet Res 2017; Mar 30 19 (03) e93
• 36 Hannig A, Kuth N, Ozman M, Jonas S, Spreckelsen C. eMedOffice: a web-based collaborative serious game for teaching optimal design of a medical practice. BMC Med Educ 2012; 12: 104
• 37 Buttussi F, Pellis T, Cabas Vidani A, Pausler D, Carchietti E, Chittaro L. Evaluation of a 3D serious game for advanced life support retraining. Int J Med Inform 2013; 82 (09) 798-809
• 38 Haubruck P, Nickel F, Ober J, Walker T, Bergdolt C, Friedrich M. et al. Evaluation of App-Based Serious Gaming as a Training Method in Teaching Chest Tube Insertion to Medical Students: Randomized Controlled Trial. J Med Internet Res 2018; 20 (05) e195
• 39 Cook NF, McAloon T, O’Neill P, Beggs R. Impact of a web based interactive simulation game (PULSE) on nursing students’ experience and performance in life support training — A pilot study. Nurse Educ Today 2012; 32 (06) 714-20
• 40 Lagro J, van de Pol MH, Laan A, Huijbregts-Verheyden FJ, Fluit LC, Olde Rikkert MG. A randomized controlled trial on teaching geriatric medical decision making and cost consciousness with the serious game GeriatriX. J Am Med Dir Assoc 2014; 15 (12) 957 e1-6
• 41 Drummond D, Delval P, Abdenouri S, Truchot J, Ceccaldi P-F, Plaisance P. et al. Serious game versus online course for pretraining medical students before a simulation-based mastery learning course on cardiopulmonary resuscitation: A randomised controlled study. Eur J Anaesthesiol 2017; 34 (12) 836-44
• 42 Katz D, Zerillo J, Kim S, Hill B, Wang R, Goldberg A, DeMaria S. Serious gaming for orthotopic liver transplant anesthesiology: A randomized control trial. Liver Transplant 2017; 23 (04) 430-9
• 43 Knight JF, Carley S, Tregunna B, Jarvis S, Smithies R, de Freitas S. et al Serious gaming technology in major incident triage training: a pragmatic controlled trial. Resuscitation 2010; 81 (09) 1175-9
• 44 Boada I, Rodriguez-Benitez A, Garcia-Gonzalez JM, Olivet J, Carreras V, Sbert M. Using a serious game to complement CPR instruction in a nurse faculty. Comput Methods Programs Biomed 2015; 122 (02) 282-91
• 45 LeFlore JL, Anderson M, Zielke MA, Nelson KA, Thomas PE, Hardee G. et al. Can a Virtual Patient Trainer Teach Student Nurses How to Save Lives-Teaching Nursing Students About Pediatric Respiratory Diseases. Simul Healthc 2012; 7 (01) 10-7
• 46 Tan AJQ, Lee CCS, Lin PY, Cooper S, Lau LST, Chua WL. et al. Designing and evaluating the effectiveness of a serious game for safe administration of blood transfusion: A randomized controlled trial. Nurse Educ Today 2017; 55: 38-44
• 47 Taekman JM, Foureman MF, Bulamba F, Steele M, Comstock E, Kintu A. et al. A Novel Multiplayer Screen-Based Simulation Experience for African Learners Improved Confidence in Management of Postpartum Hemorrhage. Front Public Health 2017; 5: 248
• 48 Sipiyaruk K, Gallagher JE, Hatzipanagos S, Reynolds PA. A rapid review of serious games: From healthcare education to dental education. Eur J Dent Educ 2018; Nov; 22 (04) 243-57
• 49 Connolly TM, Boyle EA, MacArthur E, Hainey T, Boyle JM. A systematic literature review of empirical evidence on computer games and serious games. Computers & education 2012; Sep 1 59 (02) 661-86
• 50 Sharp LA. Stealth learning: Unexpected learning opportunities through games. Journal of Instructional Research 2012; 1: 42-8
• 51 Haskell RE. Transfer of Learning: Cognition and Instruction. Elsevier Science 2000
• 52 Bloom B.S.. Taxonomy of educational objectives. Vol.1: Cognitive domain New York: McKay; 1956. p. 20-4
• 53 Gunter GA, Kenny RF, Vick EH. Taking educational games seriously: using the RETAIN model to design endogenous fantasy into standalone educational games. Educ Technol Res Dev 2008; Dec 1; 56 (5-6) 511-37
• 54 Petri G, von Wangenheim CG, Borgatto AF. MEEGA+: an evolution of a model for the evaluation of educational games. INCoD/GQS 2016 Jul;3