Strategies of Digitized Learning

Abstract

The development of digital strategies in teaching is based on the technological progress of the last decades, but also on the strong motivation to focus a didactic concept on the learning individuals. The available data of German medical faculties indicate that digital teaching concepts currently play a subordinate role in medicine in general and specifically in otorhinolaryngology. By assessing data of our own institution, we could demonstrate that the majority of medical students refer mainly to material handed out by the lecturers as single source of information for learning Otorhinolaryngology. Therefore, the application of sound digital teaching strategies provides special chances, in particular in otorhinolaryngology to cope with the excessive amount of online information from partly unclear sources.

Currently, the possible degree of digital teaching reaches from digital service supply via punctual provision of classic teaching concepts and blended learning up to completely digital curricula. The attractiveness of curricular integration of digital teaching strategies is less based on the utilization of merely technological progress, but rather on the variety of applying innovative curricula and new didactic concepts. Depending on the intended teaching purpose, the flipped classroom and the virtual reality seem to have a particularly high potential, while mobile learning is already established in individual practice. Testing and evaluating digital teaching innovations for concrete scenarios currently belongs to the most important scientific challenges of digital teaching concepts.

Today, the nationwide implementation of digital teaching in Germany is less impeded by technical conditions, but by missing financing because sponsoring is currently mainly performed with reference to concrete projects; in the context of permanent implementation, however, regular costs arise. To support these promising teaching concepts, the sponsoring of institutions for digital teaching with provision of hard- and software solutions at universities could contribute significantly. Establishing cooperation to use such digital platforms might lead to a high efficiency regarding the distribution with simultaneously profiting of savings potential.

Key words

e-learning - digital teaching - digitization - academic teaching - curriculum - medical education

1. Introduction

The development of digital teaching and according concepts is closely related with the rapid development and distribution of information technologies, in particular also with a nearly complete internet access. In order to approach the topic of digital teaching strategies, first the term of digital teaching has to be defined. Digital teaching is often also called or even used synonymously as electronic learning or e-learning. The definition, however, is not very clear which is due to the historic development from different directions. In the broadest sense, e-learning means teaching with technological support which is basically focused on the learning individual [1]. The origins date back to the 1960ies and are mainly associated with the pioneering academic teacher Patrick Suppes (Stanford University, CA, USA) and Donald Bitzer (University of Illinois, IL, USA) [1].

Suppes approached technology-supported teaching by intensively reflecting the learning-theoretical advantages of individualized computer-assisted teaching [2]. Based on these ideas, he developed a Computer Managed Instruction System as individualized instruction for learning mathematics on primary school level. Suppes considered the major advantages of using computers in the possibility of better addressing individual differences of the learning subjects and breaking up the passiveness of the auditory of lectures and presentations [2].

Bitzer and colleagues followed the technological way and established an interactive learning and teaching system called “PLATO” that provided a platform for teaching purposes to teachers based on many innovations that had been specifically developed for this tool, such as for example the programming language of TUTOR [3]. The teaching software was made available on a mainframe computer, which could be accessed by the students via decentral terminals. The motivation to develop PLATO was also triggered by the increasing number of students and aimed at reducing the costs per student contact [4]. Because of its functionality, PLATO is considered as precursor of current e-learning platforms and video conference systems [1] [3] [5] [6].

Parallel to the growing availability of computers since the 1980ies, these precedent models of computer-assisted or technology-supported teaching were expanded to multimedia interactive platforms. With increasing prevalence of the internet in the 1990ies, the local interaction between users and computers was transferred to web-based teaching contents and models of actively learning subjects were developed. Due to further technological innovation – and in particular the mobility of devices – and new possibilities of social interaction, these models were further elaborated to the current concepts of digital teaching/e-learning of our time [1] [7].

The present article will describe the current status of digital teaching strategies, elucidate current options, approaches, technologies, perspectives, and funding opportunities, and analyze them with regard to their applicability on the specific situation of medical teaching with its high percentage of practical learning. A special focus will be placed on the teaching concepts in the field of otorhinolaryngology.

2. Status of Digital Teaching in Germany

A review of digital teaching in Germany is difficult to perform because of the complexity of the terms and their heterogenic use. An indirect assessment revealed that formally 2/3^rd of German universities offer digital learning concepts [8]. However, no statement about the actual use and integration of these digital concepts can be made based on this quantitative approach. Regarding this central question, the “Hochschulforum Digitalisierung” (HFD, University Forum of Digitization), an independent institution that was founded in 2014 with funding of the Federal Ministry of Education and Research (BMBF), conducted comprehensive analyses and reports that will be cited regularly in this article [9]. As a common initiative of the german Center for University Development (Centrum für Hochschulentwicklung, CHE) and the german Conference of University Presidents (Hochschulrektorenkonferenz, HRK), the HFD pursues the objective of discussing and evaluating the impact of digitization on academic teaching. It primarily aims at implementing academic strategies, establishing competences in teaching as well as generating ideas and developments of scenarios for the future [9].

An interesting analysis of the significance of digital teaching in Germany was performed by Persike and Friedrich in 2016 on behalf of the HFD [10]. The authors evaluated a questionnaire on the topic of digitization integrated into the university ranking questionnaire of the CHE. They concluded from the assessed data that digital media were no integral part of teaching and learning at the majority of universities at the time of data collection. Due to the large sample of 27,473 students from 153 universities, further interesting aspects resulted regarding the use of digital teaching at German universities. Despite the differences between the single disciplines, an even more relevant difference of digitization between the universities themselves became obvious, which emphasizes the significance of the single university for the development and use of digital media. Additionally, an intensive analysis of the participating students was performed who were classified into different user types after evaluation of the survey. In order to simplify the analysis, Persike and Friedrich categorized 20 types of digital media into the five format types of classic digital media and communication tools (e. g. PDF documents, PowerPoint), social communication tools (e. g. blogs, chats, social networks), electronic examination systems (e-exams and e-assessments), audio- and video-based media, and interactive tools and formats. The cluster analysis of these five formats revealed four different user types of digital media: 30.2% of the students used most preferably classic digital media, 25.5% used the combination of classic media and electronic exam, 22.8% focused more on audiovisual contents beside classic media, and 21.5% were so-called “digital all-rounders” with an evenly high use of all digital learning formats. Discussing only medical disciplines, 63% of the students were found in one of the two first mentioned categories oriented at classic learning formats, while 14% learned with audiovisual media, and 23% were digital all-rounders. Within the medical disciplines, a high heterogeneity was revealed between the universities with differences of up to 25% in the frequency of using digital media. The reasons for this focus on classic digital media like PDF and PowerPoint documents could not be definitively clarified by the authors of the analysis, but they see clear hints that a regular use of digital media only occurs when they are implemented in the teaching concepts. The authors conclude that digitization in academic teaching has to be primarily fostered by the lecturers [10].

These observations are congruent with the results of an own survey with 212 students from two semesters of the Ruhr University of Bochum who completed their ENT internship in 2014–2016 [11]. 81% of the interviewed subjects stated that they used the learning material for otorhinolaryngology handed out by the lecturers. 57% of all participating students even reported that this was their only source for this discipline. These data seem to reveal that the responsibility of the lecturers in such disciplines as otorhinolaryngology has to be considered as being even higher compared to the so-called big disciplines of internal medicine or surgery. Based on this hypothesis, possibly particular chances and possibilities for the application of e-learning arise for our discipline. In the study cited above, about 90% of the students would prefer the expansion of digital learning activities.

The prevalence of digital learning tools at ENT departments of German university hospitals is currently rather low, as published by Sass and colleagues in 2017 based on an analysis of 2015 [12]. While the download of scripts or presentations as well as completing material was available at the majority of ENT departments of German university hospitals (86% and 62%, respectively), only few offers exceeded this basis. The implementation of digital media in teaching concepts requested by Persike and Friedrich seems to be sparsely distributed at ENT departments of German university hospitals.

3. Composition of a Digital Curriculum

In the general organization of digital teaching, several levels are differentiated, depending on the degree of digitization [8]. They start with classic teaching without digital media and develop via digital services and partially digitized teaching up to completely digitized studies. An overview is given in [Fig 1]. Taking a closer look to the topic, the selection of the concept or the organization is limited with regard to medical teaching. In the era of an online society, an organization without any aspect of digitization is no longer at the disposition nowadays because the basic digital infrastructure (terminal devices, networks, internet) are available nearly everywhere. Especially in the context of improving administrative services, digital services are currently regularly available as simplest level of digitization around university studies and medical education. They do not only include usual communication via e-mail, but for example also the provision with learning management systems and other central IT services (mail server, inscriptions for courses and exams, digital libraries etc.), which may support the lecturers regarding digitization of the learning material.

Fig. 1 Organizational forms of digital academic teaching [8].

Partially digitized teaching and learning may be subdivided into two main sub-aspects [8]:

1. The simplest use of e-learning is the punctual accumulation of classic teaching methods: online services are not integrated into the teaching and learning concept, but made available for selected purposes to complete the teaching/learning material. In this category, for example lecture presentations or additional material for download are provided. The classic role allocation of teaching and learning subjects is unchanged.

2. The concept of “blended learning” or “hybrid learning” enhances the level of digitization by structurally integrating the digitized contents into the curriculum and replacing currently conventional by digital material. Thus the online content is an integral part of an entire curriculum together with elements of classic academic teaching. By shifting relevant topics into the digital area, secondarily also the presence of lecturers is changed because digitally provided topics may be considered as being worked up. The percentages of both elements may vary and be adapted to the teaching and learning objective [13]. The classic role allocation of lecturers and students will be partly broken up and teaching/learning is more focused on the students. The transformation of a classic curriculum into a blended learning concept generally requires a complete re-organization of teaching.

E-learning at German medical universities is currently limited to online services or punctual accumulation concepts [14]. Due to the high percentage of practical learning in medicine, the option of completely digitized studies is not possible, however, in the context of postgraduate further education concepts (e. g. as webinars) it may be interesting [8].

How exactly such a web-based medical curriculum has to be designed, can only be described for single disciplines because of the large spectrum of medical studies, since particular requirements and aspects have to be taken into account for single disciplines. Some authors consider this increasing complexity of medicine as special chance for e-learning [14] [15]. Due to the growing complexity of the entire discipline, Prober and Khan postulate to use the chances of digitization in order to break up the missing flexibility of medical teaching [15]. Furthemore, especially the principle of flipped classroom seems to be suitable for medicine. The students should be provided with a framework curriculum serving as (online acquired) theoretical basis for further courses and seminars. Based hereon, the students should be enabled to get deeper insights into different topics according to their needs and interests.

4. Innovations

Taking into account future-oriented strategies of academic teaching, already the application of new digital technologies may seem to be a relevant innovation. However, the actual attractiveness of online learning strategies is due to the manifold options of implementing innovative curricula and new didactic concepts. In the following, selected innovations will be presented and elucidated in detail. Developing e-learning strategies is associated with the simultaneous use of different formats. The essential evaluation of those models is complex and the impact on the students is difficult to quantify. Testing and evaluating such complex blended-learning innovations for concrete scenarios belongs to the most important scientific challenges of digital teaching and e-learning concepts [16].

4.1 Flipped classroom

One of the most important learning-theoretical principles of blended learning is the so-called flipped classroom [17]. Hereby, the teaching concept is focused actively on the students; and digitization is the fundamental element of this concept. Among the different definitions of flipped classroom, also the terms of reverse, inverse, or backwards classroom are found. Structurally, the classic approach consisting of initially acquiring knowledge in courses, seminars, and lectures with personal attendance in order to subsequently do homework or transfer exercises is inversed:

The lecturers digitally provide the theoretical basics of the topic to be discussed, for example as online video. Based on this, the students prepare themselves for the inverse course/seminar at their own learning pace. Generally, the used contents may be provided in any available format. In this way, the students attending the course/seminar already have a previously acquired basic knowledge so that the course itself may focus on problem-oriented learning such as the application and increase of this basic knowledge; “homework” is then done together with the whole group [15] [18]. An overview about the structure of a flipped classroom is found in [Fig. 2].

Fig. 2 Basic principle of flipped classroom.

Several studies confirm a positive effect of flipped classrooms on the teaching and learning effectiveness and activity [19]. Numerous medical applications and analyses of this teaching/learning principle are meanwhile found in most different fields of pre-clinical [18] [20] [21] and clinical medicine [22] [23] [24] [25] [26] [27] as well as in the further education of physicians [28]. In order to implement a flipped classroom, complete re-structuring of the curriculum seems to be reasonable [15]. The creation of online material for theoretical knowledge acquisition may be extremely time-intensive. Nonetheless, the flipped classroom has particular chances to increase the learning attractiveness for the students due to the broad spectrum of design options, the freely determinable learning pace, and the possibility to achieve a higher quality level of classroom events [15] [29]. Also teaching in otorhinolaryngology could be upgraded by the high percentage of visual elements of this learning concept. The suitable percentage of courses/seminars based on the flipped classroom principle is still controversially discussed.

In an own study performed at the ORL Department of the University Hospital of Bochum, Germany, on the implementation of a flipped classroom for ORL specific courses, the hypothesis proposed by several authors could be confirmed that flipped classrooms are particularly suitable for medical teaching [11]. In the context of the internal restructuring of the course, the introduction of basic ORL specific examination strategies, refreshing of anatomic and physiological basics as well as the correct use of the instruments were included in an online course that was based on the OpenSource platform Moodle hosted by the Ruhr University Bochum. The students were instructed to work up this basic information prior to the practical course; the lecturers expected that the contents were known. Hereby it became obvious that students who intensively dealt with the online contents felt significantly better prepared for the practical course. Overall, the newly introduced teaching and learning concept was perceived positively by the students and the lecturers.

4.2 Gamification

The term of gamification is relatively new in the context of an approach of digital learning methods. The word itself describes the transformation of a non-playful context into a playful one. In the context of e-learning, it means the use of playful elements to achieve predefined learning objectives. These game-like elements may be for example a scoring system, competitions, and awards [30] [31]. Gamification may also include the simultaneous application of several playful elements, but it is differentiated from real games. In medical education, already some examples have been published on how gamification can be used. An overview of selected publications on various gamification elements can be found in [Table 1].

Table 1 Published examples of implementation of a gamification approach in medical teaching.

Publication	Approach	Gamification element	Discipline
Lamb et al. [35]	Daily medical questions on Twitter to enhance the competitive character	Incentive (Badge)	Surgery
Nevin et al. [81]	Answering questions with online ranking	Ranking Score system	Internal medicine
Chen et al. [82]	Answering questions on X-ray of the thorax within 15 s	Time limit	Radiology
Chang et al. [83]	Defined, limited allocation of roles in an emergency situation	Limitation of resources	Pediatrics

In general, this method is expected to have a high potential to improve the quality of learning by activating the intrinsic motivation of the students on the principle of the self-determination theory [30] [32]. This potential could be illustrated by publications about the competitive multiplayer game called “FoldIt”. In this context, the players contributed relevantly to the solution of complex protein folding of the HI virus [33] [34]. Depending on the design, already few technological conditions suffice for implementation of a digital gamification appraoch because for example also the use of social networks such as Twitter represents a common approach [35].

However, for the broad application of most different game-like elements – despite the described advantages – there is the question that has not been dealt with scientifically under which specific conditions gamification will be a successful learning concept [32]. Thus it may be expected that the risk for a didactic aberration is relatively high under these preconditions [32].

4.3 Game-based learning

Game-based learning describes an extended gamification approach to a real digital learning game which, regarding the content and structure, meets the pedagogic and didactic requirements of the educational objective. The objective of the game is the learning objective to be achieved [36]. Starting with quizzes up to virtual reality or complex simulation games, many different formats are possible. In 2018, Gorbanev and colleagues analyzed digital games in medical learning in the context of a review performed based on Cochrane guidelines [37]. They come to the conclusion that the mainly positive reports about medical games only have a low evidence level and that the benefit for learning is difficult to assess because of the singularity of the games. The application is mainly limited to quizzes and simulations, as there seems to be no need for more complex games in medical education [37]. Further criticism of this format states that the fundamental elements of game theory – voluntariness and absence of a purpose – are no longer valid in the context of game-based learning [36].

4.4 Records of lectures and seminars/courses

Recording and subsequent providing lecture material is a technology that is available since many years. Meanwhile automated recording systems are available in many universities and lecture halls that minimize the technological efforts and the current costs. Generally, technological processing is necessary afterwards so that the lecturers’ videos, presentations, and sound may be put together.

In Germany, no reliable data exist on the spread of lecture recordings. Even in the English-speaking countries the meaningfulness of this methods has been discussed since many years. The study situation is heterogenic; while some trials show that lecture recordings are not much used and classroom attendance is preferred [38] [39], other studies reveal a high need and use [40] [41].

The central question and interest of the lecturers of how lecture records compete and interact with visiting the “real” event seems to depend on many factors and cannot be finally answered based on the available data. The common argument that lecture records would reduce the number of students attending the classroom events does not seem to be correct. Moreover, the students seem to use the records as additional source after attending the lectures instead of replacing it [42]. An own investigation of the lecture records of the ORL Department of the University Hospital Bochum could add evidence to this hypothesis and no measurable effect could be observed regarding the number of attendances. While the number of attendances of lectures and the number of users of online records was balanced, an equal distribution of students attending the lectures frequently, sometimes, and rarely could be revealed among the intensive users of lecture records [43]. Further reasons for the lecturers’ refusal can be explained by concerns regarding the technology, the protection of intellectual property, and missing knowledge of central funding possibilities. However, the majority of the lecturers report about positive experiences after introduction of online records [42].

The advantages for students consist of a flexible working and follow-up of the lecture material at the individual learning pace. There seem to be variable percentages of user groups that consider the provision of lecture records as completion or substitution of the personal attendance [41]. In particular in cases of sickness, students with foreign mother language, conflicts with other events, or preparation of exams the possibility of lecture follow-up may be very helpful. Because of the missing innovative character, a recorded lecture replacing the “real” lecture is seen critically [36].

4.5 Mobile learning

Mobile learning is not only defined by the use of devices such as smartphones for download of learning material, but it includes all learning scenarios that are possible by the use of mobile devices independently from the location and the time [36]. With the background of broad distribution of smartphones, it may be expected that learning independently from a concrete learning context, e. g. during waiting times, in buses and trains etc. takes place. An American trial, for example, could show that more than 90% of the investigated medical student population use their smartphones for specific learning, communicating, and looking up information within their studies. They are also in favor of structured use of the medium within their courses [44]. Since a broad distribution of mobile devices cannot only be expected in students, but also in lecturers and patients, the question must be asked how the existing infrastructure may be used for teaching and learning purposes.

Beside the free, independent learning scenario, mobile learning can take place in lecture halls (e. g. live surveys) or support a teaching situation in certain learning-relevant locations (e. g. at the patients’ bedside) [36]. Masters and colleagues summarize the application of mobile learning in four interacting areas [45]:

1. The daily use of mobile devices assures the use of the same technologies for learning, teaching, and practice and vice versa, i. e. new technologies from these areas are imported into daily life.

2. In the area of mobile teaching, the expertise of mobile technologies and practical applications are developed based on the availability of mobile learning material to the students.

3. Examples and applications for mobile learning are generated from the practice, teaching and education are possible due to mobile technologies.

4. Finally, students learn to use mobile applications for the practice and share their ideas and perspectives with the teaching staff.

An overview about the concepts of Masters et al. is displayed in [Fig. 3].

Fig. 3 Model of interaction between mobile teaching, learning, and practice, adapted according to [45].

In the concrete teaching and learning situation, the principle of BYOD (“bring your own device”) may lead to technical problems because usually different operating systems and performances of the devices are found and the students are responsible for updates and functionality [36] [44]. In some scenarios, the use of centrally provided hardware such as tablets may be suitable and also has advantages regarding data safety [46]. Most trials on mobile learning focus on apps developed for the study purpose, e. g. the evaluation of a practical units or for collaborative learning [47] [48].

4.6 Social media

In the same way as mobile device, the use of social media like Facebook, Twitter, Instagram, YouTube, WhatsApp etc. is established in many people’s daily life. The provided infrastructure and the generally simple application allows social media to be used for e-learning purposes in manifold ways. Due to the large variety of options and formats, it is unclear what is actually useful. In the context of applying social media for learning purposes, the use of an already existing infrastructure may easily reach a large public [49] [50]. At best, the immediate feedback in most social media leads to vivid discussions of the available learning material [51].

In the literature, numerous applications of social media for medical learning are found on the level of academic as well as specialist-related education and learning. This apparent advantage may easily turn into a disadvantage. For example, an over-supply of information may complicate the assessment of the suitability [52]. The possibilities for every single user to provide information easily may further lead to poorly confirmed or even wrong information [53]. The development of quality criteria for specific contents is a particular challenge in this context [54].

4.7 Collaborative learning

Collaborative e-learning is mainly defined as the computer-based performance of the classic teaching and learning concept of teamwork to find a solution of a problem. The transformation of this well-known learning environment into the digital environment makes this construction independent from time and location and in this way removes this limiting factor. Classic examples are Wikis and forums.

Collaborative learning may increase the motivation for longer online courses compared to individual learning. The basic mechanisms are probably triggered by social pressure and rapid feedback of other participants [55]. Problems with collaborative learning result from complex social interactions such as for example non-attendance or predominance of single subjects. Moderating intervention of the lecturer/teacher may be reasonable [56].

4.8 Digitized reality

The umbrella term of digitized reality includes the topics of augmented reality (AR), virtual reality (VR), and simulations [36].

AR is the description of the environment with visual display of completing information. Accordingly, AR is the technically most simply implementable variant of digitized reality with various possibilities of application in teaching and learning. However, the current focus of research is rather placed on the technical implementation than on the didactic gain so that the latter one is still unclear [36]. But considering the increasing developments of AR applications for medical practice, e. g. in the context of robotic surgery [57], already allows drawing conclusions on how this technology might be applied in pre- and postgraduate teaching. Reasonable applications might be for example in surgical training by explanations/live-feedback during surgeries, or in anatomic education [58] [59] [60].

The concept of virtual reality (VR) is closely related to AR. The learning situation is completely shifted to the interactive and simulated space that is expected to display the reality. Applications of VR seem to have a high potential in medicine. For example emergency situations could be trained in virtual realities or surgical techniques might be trained. Different authors consider AR and VR as digital key technologies for the medical sector [61]. In contrast to AR or VR, simulations have been included in medical teaching earlier. They are defined as imitation/simulation of the real environment that is applied when realization is not possible because of different reasons [36]. Also for otorhinolaryngology, several practical examples exist for ear, sinus, or laryngeal surgery [62]. Despite proven advantages in studies for example regarding training in FESS [63] or handling ENT emergency cases [64], the evidence level in most studies is rather low [62]. The main argument for simulations is the possibility of training a situation that is close to reality, but that cannot be trained in reality.

4.9 Selection criteria of innovation for e-learning projects in otorhinolaryngology

Based on the heterogeneity of the above-mentioned selection of innovations, the numerous design options of digital learning projects may be illustrated. In the following, the attempt is made for ORL specific characteristics to find a further differentiation. Because of the rather poor evidence situation, in most cases no reference to scientific data can be made. Important selection criteria for innovations with a teaching/learning purpose focus on the technological competence, the complexity/efforts, cost factors, the target group, and in particular the appropriateness for the teaching purpose. A proposal of correlating these parameters is summarized in [Table 2].

Table 2 Proposed selection guide of digital innovations for a teaching concept.

Innovation	Technology	Efforts	Costs	Pre-graduate teaching		Post-graduate teaching
				Practice	Theory	Practice	Theory
Flipped classroom	+	++	++	++	+++	++	+++
Gamification	+/++	+/++	+/++	++	++	+	++
Digital games	+++	+++	++ /+++	+	+	+	+
Event recording	++	+	++	+	++	+	++
Mobile learning	++	+	+	++	++	++	++
Social media	+	+	+	+	++	+	++
Collaborative learning	+	+	+	++	++	+	++
Augmented reality	+++	++	++	+++	+	+++	+
Virtual reality	+++	+++	+++	+++	+	+++	+

(+) low / not suitable (++ ) medium / probably suitable (+++) high / promising

The relevant teaching purposes in otorhinolaryngology in Germany include the practical and theoretical training and education of medical students (pre-graduate teaching) as well as practical and theoretical further education of physicians (post-graduate teaching). This is the main difference of pre- and post-graduate teaching and learning. The first one is rather focused on theory and generally contains a short internship and longer times of lectures. The medical students as target group should acquire knowledge in the basic topics of otorhinolaryngology that qualifies them for tests and the final exam. The practical training is limited. In the context of post-graduate teaching and education, this situation is inversed: the subjects already have basic knowledge and are mainly taught practical contents (e. g. surgery courses).

Because of the rather simple technique with moderate efforts and costs, the flipped classroom seems to be suitable for most teaching purposes. Especially for pre-graduate education, several examples for application as well as advocates are found in the literature [15] [29]. In an own study, the authors could show the reasonable implementation of flipped classrooms also for the practical part of pre-graduate education [11]. Also in post-graduate education, more and more trials are found that stipulate the applicability of flipped classrooms [65] [66] [67].

As described above, only few technological conditions have to be met for education based on a gamification approach because frequently the use of the basic structures of social media is possible [35] [68]. Accordingly, the overlap with the area of social media and collaborative learning is very high, regularly these strategies are applied together [68] [69]. The didactic gain has to be considered as complex [32]. Selected published examples on gamification are summarized in [Table 1]. An interesting approach was described by colleagues of general medicine of the Charité in Berlin who tried to overcome the low interaction between colleagues of different practices by implementing case discussions in virtual groups combined with a gamification concept [69].

Also the evaluation of game-based learning has already been discussed above and reference was made to the review that was performed in 2018 based on Cochrane guidelines [37]. Because of the expected development costs and the doubtful value, the authors currently consider digital games as irrelevant for e-learning in otorhinolaryngology, independently from the intended learning purpose.

Under the aspect of flexible work-up and follow-up of theoretical information independently from the purpose, recording lectures or other teaching events is probably less relevant for courses/seminars with practical focus. Despite the low evidence in the literature, the authors emphasize the positive experiences of own lecture records at the Ruhr University Bochum [43]. Due to the fact that the central infrastructure of universities can be used, expenses and efforts can often be minimized. Shifting parts of the technological prerequisites to the users’ responsibility allows formulating the financial and technical characteristics of mobile learning in a similar way [36] [44]. The suitability of mobile learning strategies for practical as well as theoretical applications was already reported with references in chapter 4.5.

Among the described innovations, digitized reality is probably one of the most effort- and cost-intensive ones because it cannot work without complex technical basics. Because of these limitations, the reasonable application currently seems to be only possible in the post-graduate teaching and further specialization, which is congruent to the current literature. For example, the use of AR and VR applications in our discipline could support surgical training and telemedical education such as live transmissions. However, currently the low evidence in the literature contradicts to an overall application [62]. The potential of the technologies can be estimated as very high, which has already been demonstrated in some studies on surgical education in otorhinolaryngology [61] [63].

5. Financing of e-learning

The introduction of digital teaching and learning is associated with additional efforts regarding costs and staff. Accordingly, a sustainable budget and staff planning is essential for establishing and maintaining digital teaching and learning concepts [70]. Regarding the cost structure, two levels have to be differentiated that are based on different preconditions and objectives of financing. The central level defines the general financing of digital infrastructure on an inter-university or intra-university level and is a basic condition for creating decentral, specific teaching programs. The current situation of German universities leads to structural problems on both budget levels [70].

Because of the high expenses for establishing digital infrastructure, the financing is currently mostly based on project financing, which is no sustainable approach, especially regarding the current costs. Since the creation of decentral teaching/learning programs requires specific knowledge about the implementation beside the presence of digital infrastructure, there is the need of increasing and financing additional teaching staff on this second level. Since this last-mentioned aspect concerns medical teaching and thus in particular the field of ORL, also fund raising is essential for financing e-learning projects. Beside several internal funding options, also different, mostly project-related external funds are available. One protagonist in this field is the Donors' association for the promotion of humanities and sciences in Germany (Stifterverband für die Deutsche Wissenschaft) that provides for example fellowships for the promotion of digital university teaching or supports strategy development programs. Furthermore, the Federal Ministry for Education and Research (Bundesministerium für Bildung und Forschung, BMBF) promotes research projects in the field of e-learning, for example in the context of the “offensive strategy for the digital-knowledge-based society”.

Beside the acquisition of funds, possibly the implementation of cooperation and consortiums may contribute to solve the problems, in particular regarding the technical aspect [70]. In this context, an approach is imaginable that might be highly relevant also for the field of otorhinolaryngology.

6. Digital Exams

Digital exams, also called e-assessments, may be categorized based on the time of exam. Schmees and Krüger [71] classify the types as consulting (prior to higher education), diagnosing (prior to learning), formative (during the learning process), summative (after the learning process), and quality-assuring (after the course/seminar). An overview about the relevant types in medical teaching is given in [Fig 4]. According to our experience, mainly the quality-assuring type is applied in medical teaching for continuous evaluation and improvement of the courses/seminars. Furthermore, the summative format of examination corresponds basically to the classic knowledge testing in medicine, e. g. in the context of an (e-)assessment [72]. Regarding the distribution of electronic examination methods in medical disciplines in Germany, no reliable data are available. However, considering the spectrum of different examination types ([Fig. 4]), there is a broad field of possible applications with partly complex overlapping and interactions between learning and examination [72] [73] [74] [75]. In particular in the context of formative examination methods, some authors see a high potential to improve the quality of teaching and learning because the objective of exams is based on reflecting the learning outcome and to optimize teaching and learning [72] [74] [75]. One example is the so-called “Progress Test Medicine” that has been developed by the Charité in Berlin and is meanwhile established in many universities. However, it is not completely digitized [76]. The knowledge status and individual progress of medical students is regularly checked by means of 200 interdisciplinary multiple choice items, starting with the first semester of medical studies. In this way, the own knowledge can be objectively assessed and compared to others. This example makes clear that the described types of e-assessment can generally be performed even without digital support. Thus, the possible advantages and disadvantages of a digital format will be elaborated in the following.

Fig. 4 Didactic sub-categorization of e-assessment, adapted according to [71] [79].

Due to the high level of technology and digitization in medical practice, relevant contents appear that can only be assessed reasonably in a digital way, such as for example high-resolution images of microscopy or endoscopy or imaging procedures like computed tomography (CT), magnet resonance imaging (MRI), or cone beam tomography (CBT). The implementation of respective multimedia contents in an e-assessment may make those learning data also available for exams. On the basis of learning management systems of universities, the use of multimedia contents and questions seems to be suitable especially for formative examination methods. Another advantages might be the simplified, automated, and rapid evaluation as well as the transparency [77] [78].

Higher costs of digital examination methods, compared to paper-based examinations, appear mainly in the implementation phase for technological basics [79]. In particular e-tests have to meet the requirements of functional technology, safety, and operability as of the beginning. Different authors postulate that the simplification of the evaluation and maintenance finally lead to save costs or keep them on the present level because the current expenses would be significantly lower after the implementation phase [77] [79]. The legal problems of e-tests concern the inscription to the exam, the allocation of the test to the examinee, and the verification of the candidates’ identity [80]. In this regard, meanwhile numerous approaches and guidelines are available [80]. Due to the simple performance of e-assessments and the low experience of using innovative question formats, there is the risk of producing less suitable or low-quality assessments. This central problem, however, can only be solved by meeting the requirements of acknowledged guidelines [73].

It remains unclear if under certain conditions media-affine students may benefit from digital examination formats [78].

7. Conclusion

The developments in e-learning are a complex topic with multi-factor challenges. Considering the current situation of e-learning in Germany, there is a clear discrepancy especially in medicine between the digital and multimedia-based private and professional life of our society and the medical university education that is mainly based on classic strategies that are only punctually completed by digital programs. While the creation and implementation of e-learning could be simplified in the last years, also due to central services of the universities, basically mainly project-related funding is seen in this context which makes stabilization and continuation of digital contents beyond the funding period rather difficult. E-learning still has to be processed and promoted by the responsible teachers. Permanent financial provision is a conditio sine qua non. The currently deficient financial support could be at least partly compensated by cooperation between universities or respective teams within scientific societies in order to benefit from synergy effects and to initiate additional multicenter trials in this field.

Due to the high level of technology, regarding diagnostic as well as therapeutic/surgical applications, and the multitude of visual diagnoses, particular chances might appear for modern e-learning strategies in the context of otorhinolaryngology within the whole spectrum of medicine. Own investigations could show that the use of these chances for ORL seems to be reasonable, in particular based on the dependence of the students from the material provided by the lecturers/teachers. The manifold innovations in e-learning are complex with regard to their didactic assessment and require careful analysis for the intended learning purpose. Several examples of application from the literature could be shown. Regarding the applicability of these innovations within a digital overall concept for teaching/learning, however, too few reliable data are available. So, innovation in e-learning with thorough scientific evaluation must be encouraged.

By early implementing appropriate e-learning concepts for ORL in medical studies, interested students might approach the multitude and attractiveness of our discipline in a modern way and furthermore an important contribution can be made to an optimized teaching and learning for recruiting junior staff.

• Literatur

• 1 Nicholson P. A History of E-Learning. In: Computers and Education. Springer; Dordrecht: 2007: 1-11
• 2 Allen MW. Adressing diversity in (e-)learning. In: Michael Allen’s 2008 e-Learning Annual. Pfeiffer; 2008. S. 30ff
• 3 Bitzer D, Braunfeld P, Lichtenberger W. PLATO: An Automatic Teaching Device. IRE Trans Educ 1961; 4: 157-161
• 4 Bitzer DL, Johnson RL, Skaperdas D. A Digitally Addressable Random Access Image Selector and Random Access Audio System. Urbana, Ilinois: Computer-based Education Research Laboratory, University of Illinois; 1970
• 5 Hagler MO, Marcy WM. The legacy of PLATO and TICCIT for learning with computers. Comput Appl Eng Educ 2000; 8: 127-131
• 6 Suppes P. Modern Learning Theory and The Elementary-School Curriculum. Am Educ Res J 1964; 1: 79-93
• 7 Keengwe J, Kidd TT. Towards Best Practices in Online Learning and Teaching in Higher Education. 2010; 6: 9
• 8 Schmid U, Thom S, Görtz L. Ein Leben lang digital lernen – neue Weiterbildungsmodelle aus Hochschulen. Berlin: Hochschulforum Digitalisierung; 2016
• 9 Janoschka O, Friedrich J-D, Rademacher M.Das Hochschulforum. Im Internet: https://hochschulforumdigitalisierung.de/de/wir/hochschulforum
• 10 Friedrich J-D. Lernen mit digitalen Medien aus Studierendenperspektive. 2016; 45
• 11 Dombrowski T, Wrobel C, Dazert S, Volkenstein S. Flipped classroom frameworks improve efficacy in undergraduate practical courses – a quasi-randomized pilot study in Otorhinolaryngology. BMC Med Educ 2018; 18: 294
• 12 von Saß PF, Klenzner T, Scheckenbach K, Chaker A. Einsatz von E-Learning an deutschen Universitäts-HNO-Kliniken. Laryngo-Rhino-Otol 2017; 96: 175-179
• 13 Christensen CM, Horn MB, Staker H. Is K-12 Blended Learning Disruptive? An introduction to the theory of hybrids. The Clayton Christensen Institute. 2013; Im Internet: https://www.christenseninstitute.org/publications/hybrids/
• 14 Kuhn S, Frankenhauser S, Tolks D. Digitale Lehr- und Lernangebote in der medizinischen Ausbildung. Bundesgesundheitsbl 2018; 201-209
• 15 Prober CG, Khan S. Medical education reimagined: a call to action. Acad Med J Assoc Am Med Coll 2013; 88: 1407-1410
• 16 Themengruppe Curriculum Design & Qualitätsentwicklung. Design digitaler Lehr-, Lern- und Prüfungsangebote. Berlin: Hochschulforum Digitalisierung;
• 17 Johnson L, Adams Becker S, Estrada V, Freeman A. NMC Horizon Report. 2015. Austin, Texas: The New Media Consortium;
• 18 Prober CG, Heath C. Lecture Halls without Lectures — A Proposal for Medical Education. N Engl J Med 2012; 366: 1657-1659
• 19 Ramnanan C, Pound L. Advances in medical education and practice: student perceptions of the flipped classroom. Adv Med Educ Pract 2017; 8: 63-73
• 20 Morton DA, Colbert-Getz JM. Measuring the impact of the flipped anatomy classroom: The importance of categorizing an assessment by Bloom’s taxonomy.. Anat Sci Educ 2017; 10: 170-175
• 21 Whelan A, Leddy JJ, Mindra S, Matthew Hughes JD, El-Bialy S, Ramnanan CJ. Student perceptions of independent versus facilitated small group learning approaches to compressed medical anatomy education. Anat Sci Educ 2016; 9: 40-51
• 22 Lew EK. Creating a contemporary clerkship curriculum: the flipped classroom model in emergency medicine. Int J Emerg Med 2016; 9: 25
• 23 Nelson BP, Hojsak J, Dei Rossi E, Karani R, Narula J. Seeing Is Believing: Evaluating a Point-of-Care Ultrasound Curriculum for 1st-Year Medical Students. Teach Learn Med 2017; 29: 85-92
• 24 Sharma N, Lau CS, Doherty I, Harbutt D. How we flipped the medical classroom. Med Teach 2015; 37: 327-330
• 25 Sajid MR, Laheji AF, Abothenain F, Salam Y, AlJayar D, Obeidat A. Can blended learning and the flipped classroom improve student learning and satisfaction in Saudi Arabia?. Int J Med Educ 2016; 7: 281-285
• 26 Evans KH, Thompson AC, O’Brien C, Bryant M, Basaviah P, Prober C, Popat RA. An Innovative Blended Preclinical Curriculum in Clinical Epidemiology and Biostatistics: Impact on Student Satisfaction and Performance. Acad Med J Assoc Am Med Coll 2016; 91: 696-700
• 27 Grossman E, Grosseman S, Azevedo GD, Figueiró-Filho EA, Mckinley D. Flipped classroom on humanities: medicine, narrative and art. Med Educ 2015; 49: 1142
• 28 Rose E, Claudius I, Tabatabai R, Kearl L, Behar S, Jhun P. The Flipped Classroom in Emergency Medicine Using Online Videos with Interpolated Questions. J Emerg Med 2016; 51: 284-291.e1
• 29 Lin Y, Zhu Y, Chen C, Wang W, Chen T, Li T, Li Y, Liu B, Lian Y, Lu L, Zou Y, Liu Y. Facing the challenges in ophthalmology clerkship teaching: Is flipped classroom the answer? PLoS ONE 2017; 12:
• 30 Bharamgoudar R. Gamification. Clin Teach 2018; 15: 268-269
• 31 Lewis ZH, Swartz MC, Lyons EJ. What’s the Point?: A Review of Reward Systems Implemented in Gamification Interventions. Games Health J 2016; 5: 93-99
• 32 Rutledge C, Walsh CM, Swinger N, Auerbach M, Castro D, Dewan M, Khattab M, Rake A, Harwayne-Gidansky I, Raymond TT, Maa T, Chang TP.Quality Cardiopulmonary Resuscitation (QCPR) leaderboard investigators of the International Network for Simulation-based Pediatric Innovation, Research, and Education (INSPIRE). Gamification in Action: Theoretical and Practical Considerations for Medical Educators. Acad Med J Assoc Am Med Coll 2018
• 33 Cooper S, Khatib F, Treuille A, Barbero J, Lee J, Beenen M, Leaver-Fay A, Baker D, Popović Z, Players F. Predicting protein structures with a multiplayer online game. Nature 2010; 466: 756-760
• 34 Khatib F, DiMaio F, Cooper S, Kazmierczyk M, Gilski M, Krzywda S, Zabranska H, Pichova I, Thompson J, Popovi Z, Jaskolski M, Baker D. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nat Struct Mol Biol 2011; 18: 1175-1177
• 35 Lamb LC, DiFiori MM, Jayaraman V, Shames BD, Feeney JM. Gamified Twitter Microblogging to Support Resident Preparation for the American Board of Surgery In-Service Training Examination. J Surg Educ 2017; 74: 986-991
• 36 Wannemacher K, Jungermann I, Scholz J, Tercanli H, Villiez A. Digitale Lernszenarien im Hochschulbereich. Arbeitspapier Nr. 15 Berlin: Hochschulforum Digitalisierung; 2016
• 37 Gorbanev I, Agudelo-Londoño S, González RA, Cortes A, Pomares A, Delgadillo V, Yepes FJ, Muñoz Ó. A systematic review of serious games in medical education: quality of evidence and pedagogical strategy. Med Educ Online 2018; 23: 1438718
• 38 Cardall S, Krupat E, Ulrich M. Live Lecture Versus Video-Recorded Lecture: Are Students Voting With Their Feet. Acad Med 2008; 83: 1174-1178
• 39 Bacro TRH, Gebregziabher M, Fitzharris TP. Evaluation of a lecture recording system in a medical curriculum. Anat Sci Educ 3: 300-308
• 40 Johnston ANB, Massa H, Burne THJ. Digital lecture recording: a cautionary tale. Nurse Educ Pract 2013; 13: 40-47
• 41 Gupta A, Saks NS. Exploring medical student decisions regarding attending live lectures and using recorded lectures. Med Teach 2013; 35: 767-771
• 42 Kwiatkowski AC, Demirbilek M. Investigating Veterinary Medicine Faculty Perceptions of Lecture Capture: Issues, Concerns, and Promises. J Vet Med Educ. 2016;
• 43 Dombrowski T, Dazert S. Unpublished Results. 2018
• 44 Gavali MY, Khismatrao DS, Gavali YV, Patil KB. Smartphone, the New Learning Aid amongst Medical Students. J Clin Diagn Res JCDR 2017; 11: JC05-JC08
• 45 Masters K, Ellaway RH, Topps D, Archibald D, Hogue RJ. Mobile technologies in medical education: AMEE Guide No. 105. Med Teach 2016; 38: 537-549
• 46 Deutsch K, Gaines JK, Hill JR, Nuss MA. iPad experience during clinical rotations from seven medical schools in the United States: Lessons learned. Med Teach 2016; 38: 1152-1156
• 47 Chang W-H, Su Y-C, Lin AP-C, Huang M-Y. Using a mobile application to facilitate post-simulation debriefing. Med Educ 49: 1163-1164
• 48 Hsueh WD, Bent JP, Moskowitz HS. An app to enhance resident education in otolaryngology. The Laryngoscope 2018; 128: 1340-1345
• 49 Carley S, Beardsell I, May N, Crowe L, Baombe J, Grayson A, Carden R, Liebig A, Gray C, Fisher R, Horner D, Howard L, Body R. Social-media-enabled learning in emergency medicine: a case study of the growth, engagement and impact of a free open access medical education blog. Postgrad Med J 2018; 94: 92-96
• 50 Junhasavasdikul D, Srisangkaew S, Sukhato K, Dellow A. Cartoons on Facebook: a novel medical education tool. Med Educ 2017; 51: 539-540
• 51 Nettle M. Social media in medical education: Can you trust it? Yes. Emerg Med Australas 30: 416-417
• 52 Zucker BE, Kontovounisios C. It is time to improve the quality of medical information distributed to students across social media. Adv Med Educ Pract 2018; 9: 203-205
• 53 Chan R. Social media in medical education: Can you trust it? No. Emerg Med Australas EMA 2018; 30: 418-419
• 54 Paterson QS, Thoma B, Milne WK, Lin M, Chan TM. A Systematic Review and Qualitative Analysis to Determine Quality Indicators forHealth Professions Education Blogs and Podcasts. J Grad Med Educ 2015; 7: 549-554
• 55 MacNeill H, Telner D, Sparaggis-Agaliotis A, Hanna E. All for one and one for all: understanding health professionals’ experience in individual versus collaborative online learning. J Contin Educ Health Prof 2014; 34: 102-111
• 56 Saqr M, Fors U, Tedre M. How the study of online collaborative learning can guide teachers and predict students’ performance in a medical course. BMC Med Educ 2018; 18: 24
• 57 Pratt P, Arora A. Transoral Robotic Surgery: Image Guidance and Augmented Reality. ORL J Oto-Rhino-Laryngol Its Relat Spec 2018; 1-9
• 58 Zahiri M, Nelson CA, Oleynikov D, Siu K-C. Evaluation of Augmented Reality Feedback in Surgical Training Environment. Surg Innov 2018; 25: 81-87
• 59 Christ R, Guevar J, Poyade M, Rea PM. Proof of concept of a workflow methodology for the creation of basic canine head anatomy veterinary education tool using augmented reality. PloS One 2018; 13: e0195866
• 60 Küçük S, Kapakin S, Göktaş Y. Learning anatomy via mobile augmented reality: Effects on achievement and cognitive load.. Anat Sci Educ 2016; 9: 411-421
• 61 Johnson L, Adams Becker S, Estrada V, Freeman A. NMC Horizon Report. 2015. Austin, Texas: The New Media Consortium;
• 62 Musbahi O, Aydin A, Al Omran Y, Skilbeck CJ, Ahmed K. Current Status of Simulation in Otolaryngology: A Systematic Review. J Surg Educ 2017; 74: 203-215
• 63 Chan M, Carrie S. Training and assessment in functional endoscopic sinus surgery. J Laryngol Otol 2018; 132: 133-137
• 64 Smith ME, Navaratnam A, Jablenska L, Dimitriadis PA, Sharma R. A randomized controlled trial of simulation-based training for ear, nose, and throat emergencies. The Laryngoscope 2015; 125: 1816-1821
• 65 Hsu S-D, Chen C-J, Chang W-K, Hu Y-J. An Investigation of the Outcomes of PGY Students’ Cognition of and Persistent Behavior in Learning through the Intervention of the Flipped Classroom in Taiwan. PloS One 2016; 11: e0167598
• 66 Martinelli SM, Chen F, DiLorenzo AN, Mayer DC, Fairbanks S, Moran K, Ku C, Mitchell JD, Bowe EA, Royal KD, Hendrickse A, VanDyke K, Trawicki MC, Rankin D, Guldan GJ, Hand W, Gallagher C, Jacob Z, Zvara DA, McEvoy MD, Schell RM. Results of a Flipped Classroom Teaching Approach in Anesthesiology Residents. J Grad Med Educ 2017; 9: 485-490
• 67 Lucardie AT, Berkenbosch L, van den Berg J, Busari JO. Flipping the classroom to teach Millennial residents medical leadership: a proof of concept. Adv Med Educ Pract 2017; 8: 57-61
• 68 Mesko B, Győrffy Z, Kollár J. Digital Literacy in the Medical Curriculum: A Course With Social Media Tools and Gamification. JMIR Med Educ 2015; 1: e6
• 69 Dini L, Galanski C, Döpfmer S, Gehrke-Beck S, Bayer G, Boeckle M, Micheel I, Novak J, Heintze C. Online Platform as a Tool to Support Postgraduate Training in General Practice - A Case Report. GMS J Med Educ 2017; 34: Doc59
• 70 Thuy P. Finanzierung digitaler Lehre. Arbeitspapier Nr. 19. Berlin: Hochschulforum Digitalisierung, 2016 Im Internet: https://hochschulforumdigitalisierung.de/sites/default/files/dateien/HFD_AP_Nr %2019_Finanzierung_digitaler_Lehre.pdf
• 71 Schmees M, Krüger M. E-Assessments in der Hochschullehre: Einführung, Positionen & Einsatzbeispiele. 1. Aufl. Peter Lang GmbH, Internationaler Verlag der Wissenschaften; 2013
• 72 Schuwirth LWT, van der Vleuten CPM. General overview of the theories used in assessment: AMEE Guide No. 57. Med Teach 2011; 33: 783-797
• 73 Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R, Galbraith R, Hays R, Kent A, Perrott V, Roberts T. Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach 2011; 33: 206-214
• 74 Pugh D, Regehr G. Taking the sting out of assessment: is there a role for progress testing?. Med Educ 2016; 50: 721-729
• 75 Schuwirth LWT, der Vleuten CPMV. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach 2011; 33: 478-485
• 76 Osterberg K, Kölbel S, Brauns K. Der Progress Test Medizin. GMS Z Für Med Ausbild 2006; 23: Doc46
• 77 Burr SA, Chatterjee A, Gibson S, Coombes L, Wilkinson S. Key Points to Facilitate the Adoption of Computer-Based Assessments. J Med Educ Curric Dev 2016; 3:
• 78 Vogt M, Schneider S, Vogt M, Schneider S. (2009). E-Klausuren an Hochschulen. Koordinationsstelle Multimedia, JLU Gießen, 2009
• 79 Michel LP, Goertz L, Radomski S, Fritsch T, Baschour L. Digitales Prüfen und Bewerten im Hochschulbereich. Arbeitspapier Nr. 1. Berlin: Hochschulforum Digitalisierung; 2015
• 80 Vogt M, Schneider S, Vogt M.Schneider S. (2009). E-Klausuren an Hochschulen. Koordinationsstelle Multimedia, JLU Gießen, 2009
• 81 Nevin CR, Westfall AO, Rodriguez JM, Dempsey DM, Cherrington A, Roy B, Patel M, Willig JH. Gamification as a tool for enhancing graduate medical education. Postgrad Med J 2014; 90: 685-693
• 82 Chen F, Lui AM, Martinelli SM. A systematic review of the effectiveness of flipped classrooms in medical education. Med Educ 2017; 51: 585-597
• 83 Chang TP, Kwan KY, Liberman D, Song E, Dao EH, Chung D, Morton I, Festekjian A. Introducing Teamwork Challenges in Simulation Using Game Cards. Simul Healthc J Soc Simul Healthc 2015; 10: 223-226