Data Model Requirements for a Digital Cognitive Aid for Anesthesia to Support Intraoperative Crisis Management

Abstract

Objective The aim of this study is to define data model requirements supporting the development of a digital cognitive aid (CA) for intraoperative crisis management in anesthesia, including medical emergency text modules (text elements) and branches or loops within emergency instructions (control structures) as well as their properties, data types, and value ranges.

Methods The analysis process comprised three steps: reviewing the structure of paper-based CAs to identify common text elements and control structures, identifying requirements derived from content, design, and purpose of a digital CA, and validating requirements by loading exemplary emergency checklist data into the resulting prototype data model.

Results The analysis of paper-based CAs identified 19 general text elements and two control structures. Aggregating these elements and analyzing the content, design and purpose of a digital CA revealed 20 relevant data model requirements. These included checklist tags to enable different search options, structured checklist action steps (items) in groups and subgroups, and additional information on each item. Checklist and Item were identified as two main classes of the prototype data model. A data object built according to this model was successfully integrated into a digital CA prototype.

Conclusion To enable consistent design and interactivity with the content, presentation of critical medical information in a digital CA for crisis management requires a uniform structure. So far it has not been investigated which requirements need to be met by a data model for this purpose. The results of this study define the requirements and structure that enable the presentation of critical medical information. Further research is needed to develop a comprehensive data model for a digital CA for crisis management in anesthesia, including supplementation of requirements resulting from simulation studies and feasibility analyses regarding existing data models. This model may also be a useful template for developing data models for CAs in other medical domains.

Keywords

anesthesiology - checklist - data visualization - emergency treatment - software design

Background and Significance

Encouraged by promising results from nuclear power or civil aviation industries that have long embraced emergency procedure manuals or checklists to manage high-stakes, high-stress situations, interest in cognitive aids (CAs) that can counteract the potential deleterious effects of stress on human cognitive functions has grown in the field of anesthesia. Currently, crisis-related CAs commonly referred to as a “crisis checklist”, “emergency manual”, or “emergency quick reference guide” have mainly been made available in a static, paper-based format as manuals, posters, or action cards.[1] [2] [3] [4] [5]

The main function of CAs is to support trained expert teams in remembering and performing their tasks effectively rather than to help novices cope with situations beyond their expertise.[6] Ideally, CAs should provide prompts for action and evaluating critical steps during time-sensitive high-stress situations. Their goal is to offset the intense cognitive load involved in crisis management and to help translate best practices for patient care during acute events.[1] [7] [8] Further support from CAs can include current medical content and localized information (e.g., important phone numbers, depositories for critical drugs) that may help increase the speed and fluidity of crisis response.

To date, the evidence supporting the use of digital CAs instead of paper-based CAs is still inconclusive. In some simulation studies, digital CAs significantly improved decision-making and treatment,[9] [10] whereas other studies did not observe any advantage when using a screen-based CA.[11] However, it appears that familiarization and training with the CA is of greater importance for successful implementation than the format (e.g., paper or electronic) itself.[12] That said, a digital CA may have certain advantages over a paper-based version: availability, interactivity, ease of modification and update, dynamic navigation with links to other emergency scenarios, and different modes for expert and novice users. In our study, the terms “cognitive aid” and “emergency checklist” are used synonymously.

The project aimed to develop a national digital CA for crisis management in anesthesia through collaboration between human factors engineers, software developers, and anesthesiologists from different university hospitals (German Cognitive Aid Working Group, see acknowledgments). The project was funded by the Funk Stiftung.

Due to the complexity of the medical environment, the digital CA was designed to support both memory and decision-making by means of a dynamic and context-dependent presentation of information rather than a rigid checklist of prescribed action steps. To prevent poorly designed assistive technology from having a disruptive and negative effect on the user,[2] [13] [14] [15] the digital CA was developed in accordance with the User-Centered Design (UCD) Process.[16] [17] The resulting graphical user interface (GUI) builds on the strength of a digital application as compared with a paper-based CA and maintains a uniform design across all pages.

Objectives

The data structure of a digital CA has to cope with various requirements. This data structure must support the representation of critical medical information as well as the design of the digital CA and the underlying operation logic and intended features. Therefore, a requirement analysis regarding the structure of digital emergency checklists is an indispensable measure in the development process of a digital CA.

The present study describes the data model requirements for a digital CA for anesthesia to support intraoperative crisis management. It addresses the following research questions:

1. Which text elements (i.e., text modules corresponding to medical emergency information) and control structures (i.e., branches or loops within emergency instructions) must be supported by a data model for a digital CA?

2. Which properties, data types, and value ranges are required for these elements?

Methods

The data model requirement analysis for a digital CA comprised three steps:

1. Identification of general text elements and control structures in paper-based CAs relevant for a digital CA.

2. Identification of additional requirements derived from an analysis of the content, design, and purpose of a digital CA.

3. Development of a prototype data model for a digital CA derived of the identified requirements.

The study design is shown in [Fig. 1].

Fig. 1 Study design.

Identification of General Text Elements and Control Structures of Cognitive Aids

Initially, the requirements were ascertained by analyzing the format in which the emergency information is presented in paper-based CAs. This included investigating paper-based CAs for meaningful units of information, that is, text elements and control structures. For this analysis, the German Cognitive Aid Working Group provided three internationally established emergency manuals as well as two emergency checklist volumes used by national hospitals supporting intraoperative crisis management (e.g., the Emergency Manual of the Stanford Anesthesia Cognitive Aid Group[5]). Thus, a total of 85 paper-based emergency checklists, covering 43 different emergency situations, were investigated by a software engineer.

First, each individual emergency checklist was reviewed line by line, and each text element and control structure was marked as an identified emergency checklist component. Subsequently, each of these components was entered into a table row.

Second, every component (text element/control structure) was assigned to a generic term specifying its concrete function within the original text (e.g., symptoms, immediate actions, additional information, etc.). For example, sentences like “check for pulse, if no pulse, then…” and “adequate/not adequate” were both assigned to the generic term “yes/no decision”. All components that were present in two or more of the investigated emergency checklist manuals or volumes were identified.

To determine the relevance of these identified text elements and control structures of paper-based CAs for a digital CA, this list of checklist components was discussed with the German Cognitive Aid Working Group in a telephone conference.

Requirements Arising from the Content, Design, and Purpose of a Digital Cognitive Aid

Content

Subsequently, the German Cognitive Aid Working Group selected five different emergency checklists to create the content for a prototype digital CA: hypoxia (symptom), ventricular fibrillation, malignant hyperthermia, postresuscitation care, and delayed emergence from anesthesia. The building blocks for these five checklists were the general text elements and control structures previously identified from paper-based checklists as well as any other information that the clinicians considered to be relevant for a digital CA (e.g., current European Resuscitation Council guidelines).

The resulting five emergency checklists were also analyzed with regard to text elements, control structures, and the sequence of the action steps (checklist items). All components identified were compared with those derived from analyzing paper-based CAs (step Identification of General Text Elements and Control Structures of Cognitive Aids) and were listed in a requirements table. If different components showed identical properties, but only differed in their designation, they were combined and received a corresponding annotation in a separate column. For example, “symptoms” and “differential diagnoses” were both annotated with “heading for a group of items.”

Next, a separate flowchart was created using PowerPoint for each of the working groups' five emergency checklists. The five flowcharts were reviewed by the German Cognitive Aid Working Group with respect to their content and structural adequacy. Any discrepancies or errors found were discussed jointly in the project group and the documents revised accordingly. After that, the requirements table was updated by all components used in the flowcharts which were not already listed.

Design

In a following step, the requirements table was supplemented by metadata derived from the design demands of a digital CA. This metadata included information that is necessary to enable the different search options for emergency checklists identified in a UCD process.[17] The metadata supports the following five options:

• Search for emergency checklists belonging to a certain checklist category/type of situation: resuscitation (advanced cardiac life support), emergency situation, non-normal situation, and symptom

• Search for emergency checklists defined for a certain patient group (adult, parturient, and child)

• Search for emergency checklists concerning one or multiple organ systems which can be selected in an anatomical model of the body (“body navigator”)

• Search for emergency checklists assigned to one or multiple categories of the airway, breathing, circulation, disability, and exposure/environment (ABCDE) approach

• Free text search for multiple keywords assigned to an emergency checklist.

Purpose

Furthermore, requirements that are related to the purpose of our digital CA project were added to the requirements table. This includes features to share CAs between different institutions and edit locally relevant information (e.g., trade names of drugs used at the local institution, critical phone numbers, drug depositories, etc.).

Development of a Prototype Data Model Derived from the Identified Requirements

To initially verify the identified requirements, a prototype data model for a digital CA was developed. For this purpose, object-oriented modeling[18] was applied to the list of collected requirements. Similar and recurrent objects were defined as object types, that is, classes. Their characteristics were described by class attributes with respective data types and value ranges (if identifiable). The modeling process was iterative. Complex data types identified in this process that needed further specification before implementation in a data model were described with primitive data types in this prototype data model.

To demonstrate practical application, one of the five emergency checklists created as prototype content was modeled according to the prototype data model in the machine-readable data format JavaScript Object Notation (JSON). Subsequently, these data were used to display the respective emergency checklist in a digital CA prototype.

Results

General Text Elements and Control Structures of Cognitive Aids

The analysis of the reviewed paper-based emergency checklists resulted in 150 emergency checklist components, which were recorded in a table. Clustering these components according to the derived generic terms identified 24 text elements and control structures. [Table 1] lists the 21 components that were evaluated by the German Cognitive Aid Working Group as relevant for a digital CA.

Table 1 General text elements and control structures of cognitive aids

Component	Generic term
Text element
	Checklist title
	Characteristics/symptoms
	Immediate actions
	Diagnoses, differential diagnoses
	Causes
	Monitoring
	Additional information
	Drug depository
	Additional literature
	Telephone numbers (contact persons/organizational matters)
	Alternative suggestions (if it does not work this way, then try…)
	Consideration (consider/check other problems that may occur)
	Instructions for defibrillation (for ACLS)
	Instructions for resuscitation (for ACLS)
	Treatment goal (what clinical condition is to be achieved)
	Supplements, notes
	Link to another emergency checklist
	Notice for turning pages
	Note of checklist end
Control structure
	Treatment sequence (numbering)
	Decisions yes/no

Abbreviation: ACLS, advanced cardiovascular life support.

Requirements of a Digital Cognitive Aid

The analysis of the content, design, and purpose of a digital CA identified 20 requirements (18 text elements and two control structures, see [Table 2] including respective annotations).

Table 2 Requirements of a digital cognitive aid

Component	Requirement	Annotation
Text element
	Checklist title	Checklist attribute
	Last update of a checklist	Checklist attribute
	Checklist search by a category/type of situation	Checklist attribute (identified values: emergency situation, resuscitation, non-normal situation, and symptom)
	Checklist search by patient group	Checklist attribute (identified values: infant, adult, and parturient)
	Checklist search by an anatomical model of the body (“body navigator”)	Checklist attribute (identified values: brain, thyroid gland, airways, lungs, heart, circulation, abdomen, derma, and blood)
	Checklist search by ABCDE approach	Checklist attribute (identified values: A, B, C, D, and E)
	Checklist search by free text search	Checklist attribute (multiple context specific keywords)
	Different types of checklist items	Executable action step (checkable), informative step (not checkable), and recurring action step (several times checkable)
	Symptoms, immediate actions, and differential diagnoses	Heading for a group of items (identified values: symptoms, differential diagnoses, diagnostics, immediate actions, therapy, organizational matters, A, B, C, D, and E)
	“Treat acidosis,” “check for acute coronary syndrome (ACS),” and similar	Heading for a subgroup of items within an item group (variable and context specific)
	Additional information	Assigned to an item, contains pictures, tables, or literature
	Drug depository	Part of additional information
	Telephone numbers (contact persons/organizational matters)	Part of an item or additional information
	Supplements and annotations	Part of additional information
	Link to another emergency checklist	Part of an item or additional information
	Drug dosage recommendation	Part of an item or additional information
	Checklist is sharable between different instances of a digital cognitive aid	Structuring of checklists as individual objects
	Checklist can be adapted to different local conditions	Checklist attributes and items can be modified
Control structure
	Treatment sequence (numbering)	Sequence of items can be defined
	Decisions yes/no	Originating from an item

[Fig. 2] shows an excerpt of the flowchart developed for the treatment of malignant hyperthermia (adapted from German). The left column indicates whether the emergency action step can be checked off (yes/no/is recurring), the middle column contains the emergency action step itself (item), and the right column specifies any additional information for the respective emergency checklist item. The enclosing frame indicates that all displayed items pertain to an item group (here “immediate actions”). If two items are joined by an arrow, the items should be executed in sequential order.

Fig. 2 Excerpt of the flowchart developed for malignant hyperthermia (adapted from German).

Prototype Data Model

[Fig. 3] shows the derived prototype data model for a digital CA. The two large boxes correspond to the two identified classes (Checklist and Item). In each box, the class attributes are listed with their respective data types. For class attributes that only allow certain values, separate boxes contain the allowed value set.

Fig. 3 Digital cognitive aid prototype data model.

[Fig. 4] depicts a screenshot of the GUI of a prototype digital CA developed in a UCD process. The displayed emergency checklist represents a JSON object, which corresponds to the developed prototype data model. In this example, the object contains information concerning the management of malignant hyperthermia (see [Fig. 2] for comparison). The title of the checklist is displayed in the top center of the screen. The checklist items are displayed in consecutive order on the left half of the screen. The boxes to the left of each item indicate that they represent checkable items. If an item has additional information, this is marked with a small arrow at the right margin of an item. Items assigned to item groups can be accessed via tabs for each group at the top of the item list and at the right margin. In the lower left corner, the breadcrumb path indicates the checklist search history with checklist attributes.

Fig. 4 Screenshot of a digital cognitive aid prototype.

Discussion

Principle Findings

Discussion of Results

This study answered our research question by providing insight into which text elements and control structures must be supported by a data model for a digital CA for intraoperative crisis management for anesthesia. Furthermore, we determined properties, data types and value ranges required by these elements.

Our study enabled the elaboration of a comprehensive list of requirements for such a data model. The initial review process of paper-based CAs identified 19 relevant text elements and two control structures for a digital CA. The analysis of the content, design, and purpose of a developed digital CA allowed us to combine already identified elements of paper-based checklists, add additional requirements, and provide descriptive information to the list of requirements. Finally, 20 data model requirements for a digital CA were identified. Based on these requirements, a prototype data model for a digital CA was developed using object-oriented modeling. Subsequently, an emergency checklist for the treatment of malignant hyperthermia was modeled as a JSON object in accordance with the prototype data model. This data object was successfully used to integrate the respective emergency checklist in a prototype version of a digital CA. In this process, we were able to validate the developed prototype data model and the identified requirements accordingly.

The elements and structures identified by analyzing selected paper-based CAs as well as the created flowcharts revealed that several different text elements and control structures are relevant for the representation of medical knowledge in a digital CA. To enable both machine-readability as well as envisaged interactivity of respective text elements in a digital CA, critical medical information needs to be available in a structured form. Furthermore, a common structure across all emergencies within a digital CA will enhance its suitability in diverse anesthetic departments in Germany. Similar to Horng et al who developed an ontology to solve the problem of how to present the characteristic nature of an emergency department visit,[19] our study has attempted to characterize the structure of a CA for intraoperative emergencies. An ontology describes a set of domain concepts and their relations in a structured way and hence can contribute to improved domain understanding and also to promote the integration of medical knowledge into different applications. Thus, a standard representation format or even an ontology for the critical medical information of crisis management in anesthesia would be useful. However, our findings provide a basis for a common data model for digital CAs which further can also be used for other applications and systems aiming to integrate emergency checklist information.

The purpose of this study was to define requirements and structures that enable the presentation of critical medical information in a digital CA. These requirements included that every action step or decision within the digital CA has to be made by the user and that there was no need of patient data exchange with other applications, or to run in an automatized process. For example, such requirements would have to be met for the integration in a decision support system (DSS) as other authors have done with other information using the Guideline Interchange Format (GLIF)[20] [21] [22] or scoring tools.[23] The emergency checklists in this study are first and foremost used for the developed GUI and the project purpose only. However, the identified requirements for a digital CA data model are comparable with requirements that existing guideline models such as the Guideline Elements Model (GEM)[24] or GLIF meet. For example, the ActionStep element of GEM corresponds to the identified action steps of a digital CA. The ActionDescription element provides information about an action, which shows similarity with the additional information for a checklist item. The Branch_Step class of GLIF corresponds to the identified decision situation items, as it owns the attribute branches with the data type Guideline_Step and cardinality multiple. This attribute can be used to represent the determined decision situations yes and no within an item.

Furthermore, the Fast Healthcare Interoperability Resources (FHIR) HL7 standard developed resources to express plans and algorithms for clinical decision support. While the underlying digital CA focused on a simple data model for passive checklists, our developed prototype data model is compatible to FHIR. In principle, the checklist resembles a FHIR PlanDefinition and the items could be expressed as actions. However, while checklists in the long run could be translated into FHIR resources, we provided an easily manageable model suitable for passive checklists for reading reference and evaluation purposes. While the stricter specification of FHIR resources or the transfer in GEM or GLIF could benefit the definition of checklists, this was not the focus of the current evaluation project. However, such a specification should be considered as subsequent research to our study.

Discussion of Methods

The methodology chosen allowed us to address the problem of formalizing medical information for a digital CA.

A key function of the considered digital CA is to provide information that will help expert anesthesiologists to effectively manage a medical emergency. Crisis management in anesthesia, however, is not comparable to a clinical process workflow or manual procedure. In general, a process has a definite start, a sequence of action steps, different branches, and if necessary, the responsible actor for each step can be defined. An emergency situation by contrast is usually characterized by a sudden onset and rapidly changing conditions. As a result, processes that allow the user to proceed in predefined paths only are inadequate for this purpose. Furthermore, a well-designed digital CA should allow the user to enter emergency situations at different stages. For these reasons, we did not consider existing process models in more detail in our study. However, even if the data model requirements of a digital CA could be met by GEM, GLIF or a similar model, there are no reported applications in the domain of intraoperative crisis management in anesthesia. Further research would be needed to confirm the suitability of existing models or standards for this purpose. However, this was not part of our study objectives.

In addition, a formalization process in an existing data model is very time consuming and associated with errors, since it requires agreement of the domain experts on uniform data encoding. For this reason, we decided to start by identifying requirements for a data model even if this also required coordination and the absence of errors could not be guaranteed. However, we were able to show with our prototype data model that there are not many elements required for implementing data in a digital CA for intraoperative crisis management. Had we used existing data models instead this would have complicated the encoding process.

Referring to,[25] we followed a similar development process that comprised multiple iterative steps. The initial review process helped us to identify technically relevant emergency checklist elements, which then determined the extent and scope of our data model requirements. However, our literature review comprised only a limited number of paper-based emergency checklists and was performed by a scientist with a technical background. As a result, it is conceivable that some elements within the paper documents might not have been identified as relevant due to the missing medical background and that elements used in nonpaper-based emergency checklists may have remained undetected. However, an evaluation of the results by the working group as well as a comparison with published results,[26] regarding text items used in medical checklists give us reasonable grounds to assume that the list of requirements identified in our study is comprehensive.

The cooperation between domain experts and developers as well as the iterative approach was necessary to closely link medical information and technical structure. Similar approaches were applied in other studies.[20] [21] Furthermore, the different visualization methods (flowcharts, GUI) greatly enhanced the interaction and communication between anesthesiologists and software engineers. A common tool for visualizing medical guidelines as flowcharts is Protégé.[27] However, our requirements analysis showed that it is necessary to present additional information rather than just the action steps. Therefore, we chose PowerPoint to display our flowcharts. In addition, we identified metainformation that had to be visualized for evaluation but was independent of the sequence of action. This could easily be solved by presenting the respective information on additional slides of the document. The representation of item groups in the flowchart was illustrated by framing boxes as others have done.[28]

Finally, all visualization methods were evaluated by our domain experts, which ensured that the identified elements and structures met the requirements of crisis management in anesthesia.

Limitations

This study was partly dependent on results of the GUI for a digital CA developed in a UCD process and also limited by resources provided in the project. Therefore, not all requirements identified as necessary for a digital CA were specified in detail in the developed prototype data model (e.g., automatic drug dose calculation).

The visualization methods in our study were evaluated by our domain experts in an experimental setting. However, the use of a digital CA in a more dynamic, time-pressured setting (e.g., in simulation studies) will be necessary to ensure that the critical information was properly encoded and modeled. It is possible that these simulations studies will reveal additional requirements that were unknown before.

Future Research

As part of the project and an iterative UCD process, simulation studies with anesthesia teams will start in the near future. Evaluation of the video-recorded crisis scenarios as well as feedback from participants may help us identify additional requirements to the data model that could not be determined in the preceding development steps.

Based on future experience and feedback from deployment, the developed prototype data model may contribute to the elaboration of a generally valid ontology for CAs for intraoperative crisis management in anesthesia. To achieve this, the prototype data model has to be generalized.

Conclusion

The field of anesthesia has witnessed growing interest in CAs that can counteract the possible deleterious effects of stress on human cognitive function and that support efficient team work.[2] [3] [12] Currently, the majority of CAs are paper manuals, posters, or action cards. A digital version of a CA may have certain advantages over a paper-based manual, which can increase its availability, usability, and acceptance by clinicians. A prerequisite for an effective presentation of critical information is a uniform data structure, which enables consistent design and interactivity with the content. So far it has not been investigated whether standard data models can map all requirements for this purpose. The development of data models does not follow a prescribed order and is rather based on the successive iterative development steps.[25] In our study, we demonstrated a methodological approach that made it possible to define detailed requirements for describing an efficient structure of critical medical information. Future simulation-based studies of a digital CA will continue to evaluate the validity of the identified requirements. The results will further enhance the data model development of a digital CA to support intraoperative crisis management in anesthesia. It is possible that this model may also contribute to the development of data models for CAs in other domains.

Clinical Relevance Statement

A digital CA may increase safe anesthetic patient care by providing anesthesiologists with critical medical information during anesthetic crises. A uniform structure of the presented information across all emergencies listed in a digital CA may help to avoid misinterpretation and may contribute to high user acceptance. In addition, a standardized data model may possibly support the transfer of the medical information to other applications and thereby contribute to their dissemination.

Multiple Choice Questions

1. Which term is often used synonymously with “cognitive aid” in anesthesia?

   • Emergency checklist

   • Item

   • Organizational matters

   • Therapy

   Correct Answer: The correct answer is option a.

2. Which of the following attributes is part of the class Checklist as well as of the class Item in the developed prototype data model?

   • ChecklistCategory

   • Id

   • Title

   • ItemGroup

   Correct Answer: The correct answer is option b.

Conflict of Interest

None declared.

Acknowledgments

The authors would like to acknowledge the support by the German Cognitive Aid Working Group: Jan Baus, Hendrik Eismann, Oliver Happel, Axel Heller, Christopher Neuhaus, Torsten Richter, Mark Weinert, and M.S.P. (group leader). They also sincerely thank all study participants for taking the time to give their valuable feedback despite their busy clinical schedules. The present work was performed in (partial) fulfillment of the requirements for obtaining the degree “Dr. rer. biol. hum.” from Friedrich-Alexander Universität Erlangen-Nürnberg (SS).

Protection of Human and Animal Subjects

Human and/or animal subjects were not included in the project.

• References

• 1 Hepner DL, Arriaga AF, Cooper JB. , et al. Operating room crisis checklists and emergency manuals. Anesthesiology 2017; 127 (02) 384-392
• 2 Gaba DM. Perioperative cognitive aids in anesthesia: what, who, how, and why bother?. Anesth Analg 2013; 117 (05) 1033-1036
• 3 Marshall S. The use of cognitive aids during emergencies in anesthesia: a review of the literature. Anesth Analg 2013; 117 (05) 1162-1171
• 4 Marshall SD, Sanderson P, McIntosh CA, Kolawole H. The effect of two cognitive aid designs on team functioning during intra-operative anaphylaxis emergencies: a multi-centre simulation study. Anaesthesia 2016; 71 (04) 389-404
• 5 Stanford Anesthesia Cognitive Aid Group. Emergency Manual: Cognitive Aids for Perioperative Critical Events. 3rd ed. Stanford, CA: Stanford; 2016
• 6 Marshall SD. Helping experts and expert teams perform under duress: an agenda for cognitive aid research. Anaesthesia 2017; 72 (03) 289-295
• 7 Weiser TG, Haynes AB, Dziekan G, Berry WR, Lipsitz SR, Gawande AA. ; Safe Surgery Saves Lives Investigators and Study Group. Effect of a 19-item surgical safety checklist during urgent operations in a global patient population. Ann Surg 2010; 251 (05) 976-980
• 8 McEvoy MD, Hand WR, Stoll WD, Furse CM, Nietert PJ. Adherence to guidelines for the management of local anesthetic systemic toxicity is improved by an electronic decision support tool and designated “Reader”. Reg Anesth Pain Med 2014; 39 (04) 299-305
• 9 St Pierre M, Breuer G, Strembski D, Schmitt C, Luetcke B. Does an electronic cognitive aid have an effect on the management of severe gynaecological TURP syndrome? A prospective, randomised simulation study. BMC Anesthesiol 2017; 17 (01) 72
• 10 St Pierre M, Luetcke B, Strembski D, Schmitt C, Breuer G. The effect of an electronic cognitive aid on the management of ST-elevation myocardial infarction during caesarean section: a prospective randomised simulation study. BMC Anesthesiol 2017; 17 (01) 46
• 11 Watkins SC, Anders S, Clebone A. , et al. Mode of information delivery does not effect anesthesia trainee performance during simulated perioperative pediatric critical events: a trial of paper versus electronic cognitive aids. Simul Healthc 2016; 11 (06) 385-393
• 12 Goldhaber-Fiebert SN, Howard SK. Implementing emergency manuals: can cognitive aids help translate best practices for patient care during acute events?. Anesth Analg 2013; 117 (05) 1149-1161
• 13 Peute LWP, Jaspers MWM. The significance of a usability evaluation of an emerging laboratory order entry system. Int J Med Inform 2007; 76 (2-3): 157-168
• 14 Han YY, Carcillo JA, Venkataraman ST. , et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116 (06) 1506-1512
• 15 Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004; 11 (02) 104-112
• 16 International Organization for Standardization. ISO 9241–210:2010–03, Ergonomics of human-system interaction: part 210: human-centred design for interactive systems 2010. Geneva: Available at: https://www.iso.org/standard/52075.html . Accessed February 7, 2020
• 17 Schild S, Sedlmayr B, Schumacher A-K, Sedlmayr M, Prokosch H-U, St Pierre M. ; German Cognitive Aid Working Group. A digital cognitive aid for anesthesia to support intraoperative crisis management: results of the user-centered design process. JMIR Mhealth Uhealth 2019; 7 (04) e13226
• 18 Jacobson I, Christerson M, Jonsson P, Overgaard G. Object-Oriented Software Engineering: A Use Case Driven Approach. 1st ed. London, United Kingdom: Pearson; 1992
• 19 Horng S, Greenbaum NR, Nathanson LA, McClay JC, Goss FR, Nielson JA. Consensus development of a modern ontology of emergency department presenting problems-the hierarchical presenting problem ontology (HaPPy). Appl Clin Inform 2019; 10 (03) 409-420
• 20 Patel VL, Allen VG, Arocha JF, Shortliffe EH. Representing clinical guidelines in GLIF: individual and collaborative expertise. J Am Med Inform Assoc 1998; 5 (05) 467-483
• 21 Choi J, Currie LM, Wang D, Bakken S. Encoding a clinical practice guideline using guideline interchange format: a case study of a depression screening and management guideline. Int J Med Inform 2007; 76 (Suppl. 02) S302-S307
• 22 Boxwala AA, Peleg M, Tu S. , et al. GLIF3: a representation format for sharable computer-interpretable clinical practice guidelines. J Biomed Inform 2004; 37 (03) 147-161
• 23 Shelov E, Muthu N, Wolfe H. , et al. Design and Implementation of a Pediatric ICU Acuity Scoring Tool as Clinical Decision Support. Appl Clin Inform 2018; 9 (03) 576-587
• 24 Shiffman RN, Karras BT, Agrawal A, Chen R, Marenco L, Nath S. GEM: a proposal for a more comprehensive guideline document model using XML. J Am Med Inform Assoc 2000; 7 (05) 488-498
• 25 Noy NF, McGuinness DL. Ontology development 101: a guide to creating your first ontology. 2001 . Available at: http://protege.stanford.edu/publications/ontology_development/ontology101.pdf . Accessed February 7, 2020
• 26 Burian BK, Clebone A, Dismukes K, Ruskin KJ. More than a tick box: medical checklist development, design, and use. Anesth Analg 2018; 126 (01) 223-232
• 27 Musen MA. ; Protégé Team. The Protégé project: a look back and a look forward. AI Matters 2015; 1 (04) 4-12
• 28 Boxwala AA, Tu S, Peleg M. , et al. Toward a representation format for sharable clinical guidelines. J Biomed Inform 2001; 34 (03) 157-169