Development of a Perioperative Medication-Related Clinical Decision Support Tool to Prevent Medication Errors: An Analysis of User Feedback

Abstract

Objectives Medication use in the perioperative setting presents many patient safety challenges that may be improved with electronic clinical decision support (CDS). The objective of this paper is to describe the development and analysis of user feedback for a robust, real-time medication-related CDS application designed to provide patient-specific dosing information and alerts to warn of medication errors in the operating room (OR).

Methods We designed a novel perioperative medication-related CDS application in four phases: (1) identification of need, (2) alert algorithm development, (3) system design, and (4) user interface design. We conducted group and individual design feedback sessions with front-line clinician leaders and subject matter experts to gather feedback about user requirements for alert content and system usability. Participants were clinicians who provide anesthesia (attending anesthesiologists, nurse anesthetists, and house staff), OR pharmacists, and nurses.

Results We performed two group and eight individual design feedback sessions, with a total of 35 participants. We identified 20 feedback themes, corresponding to 19 system changes. Key requirements for user acceptance were: Use hard stops only when necessary; provide as much information as feasible about the rationale behind alerts and patient/clinical context; and allow users to edit fields such as units, time, and baseline values (e.g., baseline blood pressure).

Conclusion We incorporated user-centered design principles to build a perioperative medication-related CDS application that uses real-time patient data to provide patient-specific dosing information and alerts. Emphasis on early user involvement to elicit user requirements, workflow considerations, and preferences during application development can result in time and money efficiencies and a safer and more usable system.

Keywords

medication safety - clinical decision support - patient safety - user-centered design - user feedback

Background and Significance

Medication-related incidents, including medication errors (MEs) and adverse medication events (AMEs), are common in the perioperative setting.[1] [2] [3] Perioperative ME rates (4–11%)[1] [2] are consistent with ME rates in other hospital areas, such as inpatient wards (5–19%)[4] [5] [6] [7] and outpatient clinics (7–12%).[8] [9] [10] However, 10 to 13 medications are administered per operation,[1] [11] resulting in a high percent of operations involving medication-related incidents.[1] Almost half of medication-related incidents involve observed patient harm and the remainder have the potential for harm.[1] [10] [12] Thus, preventing MEs in the operating room (OR) is of great public health importance and has become a priority locally, nationally,[13] [14] and internationally.[15]

Medication use in the OR today presents particular patient safety challenges because it often bypasses standard safety checks, such as electronic order entry with decision support and nursing double checks prior to medication administration. In fact, the OR is one of the few locations where every step of the medication use process (medication selection, dispensing, preparation, administration, documentation, and monitoring) is typically completed by a single clinician (the anesthesia clinician), without safety checks by a second clinician or by clinical decision support (CDS) with alerts to warn of MEs.

Two main features of the OR limit the use of existing medication-related electronic CDS. First, there are typically no prospective medication orders in the OR. Documenting medication in the anesthesia information management system (AIMS) functions as both a retrospective order and documentation that the medication was administered. Second, surgical patients are often among the highest acuity patients in the hospital, and due to the nature and potency of medications administered in the OR, patients' conditions can quickly change while under anesthesia.

User-centered system design is an iterative approach that involves users from initial concept development through implementation, with the goal of making systems usable by focusing on user requirements, and by applying human factors/ergonomic principles and usability techniques.[16] [17] While there is widespread agreement that user-centered design should be used in electronic health record (EHR) development and if it is incorporated the EHR is more likely to be effective, user-centered design principles are often not followed.[18] There have been broad concerns about EHR usability and in a recent study, U.S. physicians gave EHRs an “F” rating on usability.[19]

Certain EHR functionality (such as CDS and alerts) has been incorporated less frequently in the OR, in part because of the pace of care and the nontraditional medication use process. CDS has been shown to reduce MEs in many non-OR locations, including inpatient wards,[20] [21] [22] outpatient clinics,[23] emergency rooms,[24] intensive care units,[25] and long-term care facilities.[26] Intraoperative medication-related CDS on the market currently is often limited to simple reminders (e.g., to redose antibiotics, monitor blood glucose, or administer postoperative nausea prophylaxis).[27] [28] [29]

Poor CDS design can lead to negative consequences, such as AMEs and increased provider burden.[31] [32] [33] While there is a paucity of data on user-centered design of intraoperative CDS applications, the usability of CDS applications has been well studied outside of the OR.[33] [34] [35] To maximize the safety benefits of intraoperative CDS as it becomes more robust and incorporates increased functionality, we believe it must be built using user-centered design.

Objectives

The purpose of this paper is to describe the development and analysis of user feedback for a robust, real-time medication-related CDS application for use in the OR. The user feedback will be used together with future summative usability testing to inform the design of the CDS application.

Methods

The study was conducted at a 1,046-bed tertiary care academic medical center that performed approximately 150 operations per day in 58 ORs excluding off-site anesthetizing locations. There were 276 anesthesia clinicians, including 115 anesthesiologists, 56 certified registered nurse anesthetists (CRNAs), and 105 house staff. The anesthesia clinicians used an electronic AIMS (Epic, Verona, Wisconsin, United States) to document patient demographics, vital signs, medications, perioperative events, and related procedures. Each OR had a barcode-assisted syringe labeling device (Codonics, Middleburg Heights, Ohio, United States) that printed color-coded syringe labels at the time of syringe preparation. Prefilled syringes were also available for select medications. This study was approved by our Institutional Review Board and the requirement for written consent was waived.

We developed a perioperative CDS application with a multidisciplinary team, including individuals with expertise in clinical anesthesia, human factors, informatics, statistics, system design, usability, and programming. The CDS application has been implemented in the ORs at our institution. Application development occurred in the following six phases: (1) identification of need, (2) alert algorithm development, (3) determination of technical requirements, (4) analysis of user feedback, (5) system design, and (6) user interface design.

Identification of Need

We conducted a prospective observational study of 277 operations (with >3,600 medication administrations) at our center and found that 5.3% of medication administrations involved an ME and/or AME.[1] We found that more than half of the MEs, nearly all of the AMEs, and one-third of the potential AMEs (near misses) had the potential to be prevented by CDS with alerts.[1]

Alert Algorithm Development

We mapped each ME and/or AME in our observational study[1] to a CDS alert rule that would have prevented it. For example, if an AME involved hypotension, defined as a mean arterial pressure (MAP) <55 mmHg, the corresponding CDS rule was “alert if MAP <55 mmHg.” We then used the extensively studied RAND-UCLA Delphi's Method[36] [37] [38] [39] to achieve consensus among 25 international experts on the most important CDS rules to include. The underlying goals were to (1) minimize the number of alerts, especially hard-stop alerts, to prevent alert fatigue; and (2) include those alerts with the highest potential to prevent serious and life-threatening MEs.

This process resulted in two main categories of alerts: (1) medication-triggered alerts and (2) reminder alerts. Medication-triggered alerts display prior to medication administration if the CDS algorithms detect a ME, such as a life-threatening medication allergy, when the medication syringe is scanned. Reminder alerts are triggered by real-time, intraoperative patient data, including vital signs, laboratory values, and ventilator data. If the CDS algorithms detect a recommended action that has not yet been performed, a reminder alert will display. For example, if the MAP drops below a critical value (that is individualized based on patient comorbidities and baseline blood pressure) for an extended time, an alert will display to remind the clinician to address the blood pressure.

We used a Condorcet ranking method called the crowd ranking kit[40] [41] to assign alerts to be either interruptive (critically important alerts that pop up over the standard EHR workflow) or integrated into the workflow, such as customized default medication doses, based on the patient's weight, age, and/or renal function.

Determination of Technical Requirements for Perioperative Clinical Decision Support

In consultation with our informatics specialists (K.C.N., A.B., W.J.G., and D.W.B.), we identified three technical requirements for implementation of the perioperative CDS alert algorithms.

Real-Time Functionality

It is especially important in the fast-paced, rapidly changing OR environment that CDS alerts be able to incorporate real-time patient data such as vital signs and medications being administered. For example, CDS should capture whether medications are incompatible with a patient's heart rate in real-time, or whether there may be a drug-drug interaction with a prior medication that was administered seconds ago. Most AIMS do not have access to real-time vital sign data, and instead typically collect these data averaged over 1 minute or more,[30] which is too slow for important CDS functionality in the OR. For example, 1-minute temporal resolution data could lead to alerts falsely firing (e.g., based on a heart rate value from 1 minute ago that has since been corrected). Thus, it is necessary to build the CDS algorithms outside of the AIMS to achieve real-time functionality.

Individualized Alerts

Delivery of anesthesia is often tailored to patient and provider preferences. Anesthesiologists may titrate doses to effect while at the same time considering patients' physiologic limitations such as renal insufficiency, pain, or coronary artery disease, making too much standardization of practice challenging and undesirable. Medication-related CDS alerts in the OR must be tailored to a patient's baseline physiology, incorporating variables such as baseline blood pressure and renal function. This cannot be done with alerts that have static triggers. For example, a blood pressure that is too low for one patient may be acceptable for another. An antibiotic dose for a patient with normal renal function may be too high for a patient with renal insufficiency.

Workflow Change

Anesthesia clinicians typically document medications in the AIMS after administration. For intraoperative CDS to be maximally effective, initiation of medication documentation should instead occur immediately prior to medication administration, allowing an alert to be displayed when necessary to prevent a ME prior to the medication being administered ([Fig. 1]). While this workflow change can be achieved with manual documentation, it may be facilitated by point-of-care barcode scanning of syringe labels immediately prior to medication administration. Point-of-care barcode scanning may also improve efficiency and automate workflow by launching the medication dosing window, so that the clinician simply inputs the dose instead of also manually searching for the medication dosing window. While inputting a predicted dose prior to administration allows for additional CDS functionality, such as wrong dose alerts, it is not always possible to predict final doses when medications are being titrated. Thus, when necessary, final doses can be documented after administration. Also, if multiple medications are being given in a short period of time (e.g., during induction), the workflow should allow for them to be scanned in quick succession, so that the clinician can go back and confirm the doses when time permits. The initial scan of the syringe label barcode (without a final dose) allows for the majority of CDS functionality, such as age-, weight- or renal-based dosing suggestions, and alerts that are independent of dose such as severe medication allergies or drug-drug interactions. The introduction of processes to automate workflow and improve efficiency such as point-of-care barcode scanning, along with strong communication, user feedback, and champions (trusted peer advocates and coaches for change) can facilitate this workflow change.

Fig. 1 Example medication bolus workflow with and without CDS application. AIMS, anesthesia information management system; CDS, clinical decision support.

Analysis of User Feedback

We conducted iterative user-centered design activities including group and individual design feedback sessions with front-line clinician leaders and subject matter experts to gather information about user requirements for the alert content and usability of the CDS application interface. Participants were clinicians who provide anesthesia (including attending anesthesiologists, CRNAs, and house staff), OR pharmacists, nurses, and patient-safety experts.

We began each group and individual design feedback session by providing background on perioperative MEs and describing the CDS application functionality. We then displayed mock-ups of sample alerts that were drawn in Microsoft PowerPoint (Redmond, Washington, United States). An anesthesiologist (K.C.N.) and a user-centered design expert (P.M.G.) developed a semistructured guide (Appendix A), including open-ended questions to prompt discussion about the workflow and clinical usefulness of the alerts, and elicit feedback on the alert content and user interface to facilitate user acceptance. We incorporated feedback until we reached information saturation, the point at which we were no longer gaining new information or feedback from successive design sessions.

Analyses: descriptive statistics are reported as counts with percentages, using Microsoft Excel. Interview notes were analyzed using applied inductive thematic analysis[42] [43] to develop a list of themes (comments or ideas) to describe the user feedback. Reviewer bias was minimized by having two independent study staff (K.C.N. and P.M.G.) iteratively review the notes and group similar themes together. In parallel with design sessions, the reviewers met weekly to discuss and iteratively modify the themes, identify emerging themes, and further delineate the relationships among them until they reached consensus on all themes. This manuscript adheres to the Standards for Reporting Qualitative Research (SRQR) guidelines.[44]

System Design

To maximize generalizability and to allow for real-time functionality (see “Determination of Technical Requirements for Perioperative Clinical Decision Support” section), we built the CDS application outside of our AIMS. The clinician launches the CDS instantaneously via a button in the patient's chart, and the CDS runs behind the EHR/AIMS (on the same computer monitor), receiving patient and provider context from the EHR. The CDS inputs and outputs are illustrated in [Fig. 2]. The CDS consists of the following three modules:

Fig. 2 CDS application inputs and outputs.

1. Streaming rule engine: the streaming rule engine creates alerts based on real-time data from the vital sign monitor and ventilator.

2. Synchronous rule engine: the synchronous rule engine receives inputs from the EHR and creates alerts when necessary at the start of a case (e.g., missing laboratory tests when indicated) and in response to intraoperative medications administered or laboratory results. The engine extracts patient data from the EHR via the Health Level 7 (HL7) Fast Healthcare Interoperability Resources (FHIR) standards[45] at the start of each case (e.g., patient demographics) and during the case in real-time (e.g., intraoperative laboratory values). The engine also extracts medication data from barcode scans. Using an off-the-shelf barcode scanner, clinicians scan the barcode on any type of syringe label (manufacturer-applied label, pharmacy-applied label, etc.) immediately prior to medication administration. The scan sends the medication's National Drug Code which serves as a universal medication identifier to the rule engine and also launches the CDS medication dosing window.

3. Web application: the web application was developed and refined iteratively based on user feedback. It sits behind the AIMS on the existing anesthesia workstation computer monitor, and displays windows on top of the AIMS as needed to provide two functions: (1) it presents alerts to the user in real-time, and (2) it presents medication dosing windows that are populated with individualized dosing information (e.g., renal-, age- or weight-based, when applicable), so that the provider just confirms the desired dose either before or after administering the medication. While inputting a predicted dose prior to administration allows for additional CDS functionality such as wrong dose alerts, the initial barcode scan (even without inputting a final dose) allows for a majority of the CDS functionality such as dose suggestions and critical alerts that are independent of dose such as severe medication allergies or drug-drug interactions. After the final medication dose is entered, the medication data are sent to the patient's record in the AIMS for automated documentation in real-time, eliminating the need to manually document the medication in the AIMS.

User Interface Design

The CDS application user interface includes medication dosing windows and alert windows. Medication dosing windows display the medication name and have fields for data input by the user, including dose (with patient-specific quick-choice buttons), route, and administration time. Patient-specific default values are included for all fields ([Fig. 3]).

Fig. 3 Medication dosing window: ketorolac.

The CDS application also generates alert windows, when applicable, for critical alerts (e.g., wrong medication or significant medication allergy), and reminders (e.g., to intervene when indicated by patient monitor data or to check glucose when necessary). The initial alert design was generated from requirements gathered during the Delphi process (see “Methods: Alert Algorithm Development” section) and with our usability specialist (P.M.G.) based on best practices for usability of CDS alerts such as using concise and consistent language, providing actionable tools, including relevant contextual information within the alert, and delivery at the time of decision-making.[33] [47] [48] An example alert window is shown in [Fig. 4]. Upon receiving an alert, the provider may decide to accept the alert and revise the action that generated the alert or override the alert and continue with the planned action. Alert overrides trigger an accountable justification prompt which requires the user to indicate a reason for override by selecting from a dropdown menu of reasons or entering a new reason. Collecting and analyzing data on provider response to alerts is essential to keeping alerts relevant, useful, and safe. Alerts with high levels of appropriate overrides can be modified or deleted; those with high levels of inappropriate overrides are good targets for provider educational interventions; and those with high levels of acceptance can remain.

Fig. 4 Alert window.

Results: User Feedback Analysis

We performed two group and eight individual design feedback sessions. In total, there were 35 participants ([Table 1]). We identified 20 feedback themes corresponding to 19 CDS design changes. Examples of the feedback and our resulting design changes are shown in [Table 2]. The first group design feedback session consisted of a 1-hour session with 21 participants who were members of the department's Quality and Safety Improvement Committee. Participants included eight attending anesthesiologists, four anesthesia house staff, two CRNAs, five patient safety experts, one pharmacist, and one OR nurse. Feedback themes included the following: users did not want alerts with hard stops that force them to take a particular action, users wanted a mechanism to override alerts, and users discussed the implications and the best practices for achieving the required workflow change (initiation of the medication documentation process immediately prior to medication administration instead of after medication administration).

Table 1 Participant characteristics

	Group session 1	Group session 2	Individual design sessions	Total
Number of participants	21	6	8	35
Female n (%)	13 (61.9)	4 (66.7)	1 (12.5)	18 (51.4)
Breakdown by specialty n (%)
Anesthesiologist	8 (38.1)	2 (33.3)	6 (75.0)	16 (45.7)
House staff	4 (19.0)	1 (16.7)	–	5 (14.3)
Certified registered nurse anesthetist	2 (9.5)	–	–	2 (5.7)
Operating room nurse	1 (4.8)	1 (16.7)	–	2 (5.7)
Pharmacist	1 (4.8)	1 (16.7)	1 (12.5)	3 (8.6)
Patient/medication safety expert	5 (2.4)	1 (16.7)	1 (12.5)	7 (20.0)

Table 2 Example feedback themes and corresponding system design changes

	Example feedback theme	Design change
First group session	Users dislike being forced to take a particular action	Included a “Reject Suggestion” button for all alerts
	There may be valid reasons for overriding an alert	Added accountable justification prompt, which requires users to select a reason for rejecting an alert by choosing from a dropdown menu of acceptable reasons, or entering a new reason as free text
	Workflow change (required to document medications immediately prior to administration instead of immediately after administration) might be challenging to achieve	Incorporated point-of-care barcode scanning of syringe label to ease documentation workflow
Second group session	Users may want to change medication units	Made unit field editable
	Users may want to edit time of medication administration	Made time field editable
	With increased use of sugammadex for reversal of neuromuscular blockade, alerts about potential prolonged response to neuromuscular blockers may be less relevant	Deleted alert about prolonged response to neuromuscular blockade in patients with renal failure
Individual design sessions	Users may want even easier access to the alert rationale	In the reference link, provided a summary of the relevant text from the reference, in addition to providing the reference citation

The second group design feedback session consisted of a 90-minute session with six participants including one OR pharmacist, two anesthesiologists, one OR nurse, one anesthesia resident, and one usability expert. Feedback themes included the following: users wanted the ability to edit values such as the time, units, and computed baseline values (e.g., baseline blood pressure); and users wanted to know the underlying logic and rationale behind alerts. For example, an alert that simply states that “MAP is too low” may be viewed differently if presented along with the rationale “Patient has preexisting hypertension with a baseline MAP of 150 mm Hg.”

Participants in the individual design sessions included six anesthesiologist front-line clinician leaders and two subject matter experts (one pharmacist and one medication safety expert). Their feedback themes included the following: users preferred to receive more information with each alert, and users wanted hard stops alerts only when necessary, and no duplicate alerts.

Discussion

We incorporated a user-centered design process to build a perioperative medication-related CDS application that uses real-time patient data to generate individualized alerts, when applicable. Through group and individual design sessions, we identified 20 feedback themes, corresponding to 19 system changes. Our hope is that incorporating these changes will improve the usability of the CDS application, leading to greater user acceptance. The main themes that we identified for user acceptance were using hard stops only when necessary, providing as much information as feasible about the rationale behind alerts and the clinical context, and allowing users to edit fields such as units, time, and baseline values. Some themes are easily addressed. Providing information about the alert rationale is straightforward and aligns with the important goals of achieving data integrity and transparency. Allowing users to edit fields is also straightforward, though it is essential to clearly indicate the units for a field that will be used in a calculation such as weight. While hard stops should be used extremely infrequently, they can be necessary, for example, to avoid giving a potentially lethal dose of potassium.

While there is a paucity of data on user-centered design of intraoperative CDS applications, Schild and colleagues describe the importance of a user centered design process for a digital cognitive aid that provides checklists for intraoperative crises.[49] The usability of CDS applications has been well studied outside of the OR, with similar findings to ours, including recommendations to design alternatives to hard stops, provide the rationale behind alerts, and allow modifications to orders.[33] [34] [35] Horsky and colleagues conducted a targeted review of the best practices for medication-related CDS outside the OR and found that user-centered, iterative design is critical, and recommendations included tiering alerts by severity to avoid excessive use of interruptive alerts, including relevant patient information on dosing screens and using concise text with justifications.[33] The themes we identified also align closely with Nielsen's Usability Heuristics, which are principles that are rooted in human factors engineering and have been extensively used to evaluate the usability of software and web applications.[50] [51] [52] For example, our users' preferences for editing fields and avoiding hard stops align with Nielsen's heuristic of User Control and Freedom. Our users' feedback about workflow change aligns with Nielsen's Heuristic of Flexibility and Efficiency of Use, which is an especially important heuristic in perioperative settings where tools with poor usability may take focus away from the patient.

Real-time medication-related CDS in the OR has been difficult to achieve to date.[27] [30] [53] [54] This is one of the first applications of real-time, individualized CDS for medication administration in the OR,[29] [30] which will likely improve patient safety not only by preventing MEs before they reach the patient but also by reducing the harm of nonpreventable AMEs through alerts to limit the duration of serious hemodynamic abnormalities. While not yet evaluated in the OR, CDS has been shown to prevent MEs in inpatient and outpatient settings, when it has been designed well and human factors issues have been addressed.[20] [55]

Medication-related CDS in the OR involves a change in the current medication use workflow and any such change must be addressed carefully. Specifically, medication documentation must be initiated immediately prior to medication administration, rather than after administration. By scanning a medication syringe prior to administration, the following two important outcomes can be achieved: (1) automation of the documentation process by scanning instead of manually searching for medications, and (2) allowing CDS functionality to warn of serious MEs before they reach the patient. Given this workflow change, the design of the alerts, their integration into the workflow, and their value to clinicians in terms of reducing cognitive load and improving efficiency are critical to achieving user satisfaction. Changes we made to the initial CDS design based on user feedback were not extensive but they may still have a substantial effect on clinicians' overall willingness to accept the alerts. Alert fatigue represents a high priority concern, and in a critical setting such as the OR (where the “noise” today is already high), clinician feedback should be incorporated into alert design to limit extraneous information and provide actionable tools. Poorly defined user requirements early in software development have been identified as a major contributor to project failure.[56] [57] [58] Health care has been slow to adopt a user-centered design approach, but efforts in the last decade to focus on usability have led to a greater understanding of its importance.[16] [59] [60] A greater emphasis on early user involvement to elicit user requirements and preferences prior to development can result in time and money efficiencies, as well as a safer and more usable system.[18] [61] [62]

Limitations

This study has limitations. First, our participants are members of the anesthesia department at a large tertiary care academic medical center. Their preferences and feedback may not reflect clinicians who work at smaller centers or private practice. Second, the study site is an advanced OR environment with an electronic AIMS and barcode-assisted medication labeling device, which may not be available at other hospitals. To facilitate its generalizability, we designed the CDS to recognize the barcode on any syringe label (manufacturer-applied, OR pharmacy-applied, or barcode-assisted syringe labeling device). The CDS also allows for manual medication entry via a search function (similar to standard AIMS) instead of barcode scanning. Finally, while the CDS application is built on an open-source platform using FHIR international data exchange standards to maximize generalizability, some local implementation and testing of the interfaces may be required at other institutions, as is the case with any clinical application.

Conclusion

We incorporated a user-centered design process to build a perioperative medication-related CDS application that uses real-time patient data to generate individualized alerts, when applicable. Through group and individual design sessions, we identified the following three main requirements for user acceptance: (1) using hard stops only when necessary; (2) providing as much information as feasible about the rationale behind alerts and the clinical context; and (3) allowing users to edit fields such as units, time, and baseline values. Our use of a user-centered design process is likely to improve the usability of the novel CDS application, leading to greater user acceptance. Future work will focus on summative usability testing of the CDS application in a clinical setting, followed by measuring the impact of the CDS application on perioperative ME and AME rates.

Clinical Relevance Statement

Meaningful use of perioperative clinical decision support (CDS) has the potential to reduce the incidence of perioperative MEs[1] including dosing errors, errors of omission, monitoring errors, and wrong medication errors. In fact, our prior work has demonstrated that perioperative CDS with alerts has the potential to eliminate more than half of MEs, nearly all AMEs and one-third of potential AMEs (near misses).[1] The incorporation of user-centered design is critical to achieve user acceptance of perioperative medication-related CDS in general,[63] and we suspect this will also hold true in the perioperative setting, facilitating a reduction in the incidence of MEs, and making surgery and anesthesia safer for patients.

Multiple Choice Questions

1. Based on the user feedback presented in this study, the authors determined which of the following features of clinical decision support (CDS) systems are important:

   • Having many hard stops

   • Uneditable time fields

   • Alerts containing the underlying logic and rationale behind the recommendations

   • Duplicate alerts as an additional safety precaution

   Correct Answer: The correct answer is option c. Choice a is incorrect because user feedback indicated that users did not want CDS to contain hard stops that force them to take a particular action. Hard stops may be necessary at times but should generally be avoided. Choice b is incorrect because the authors found users wanted to be able to edit the time fields, as well as units and baseline values. Answer choice d is incorrect because users did not want duplicate alerts, which can lead to alert fatigue. Answer choice c is correct because, alongside having a reference link, users indicated that they wanted to know the alert rationale.

2. Which of the following did the authors identify as a technical requirement of clinical decision support (CDS) systems in the operating room (OR)?

   • Real-time functionality

   • 1-minute temporal resolution data

   • Alerts that are the same for every patient

   • A similar workflow to existing AIMS

   Correct Answer: The correct answer is option a. Choice a is correct because in the fast-paced, rapidly changing OR environment, CDS alerts should incorporate real-time patient data. Choice b is incorrect because one-minute temporal resolution data could lead to alerts falsely firing. Choice c is incorrect because CDS alerts in the OR should be tailored to a patient's baseline physiology, incorporating variables such as baseline blood pressure and renal function. Choice d is incorrect because anesthesia clinicians typically document medications in the AIMS after administration. For the CDS to be most effective, clinicians should initiate documentation of medications before administration, so that alerts can prevent medication errors before they reach the patient.

Conflict of Interest

K.C.N. receives author royalties from UpToDate, Inc. (Waltham, Massachusetts, United States). A.B. is a stockholder in Elimu Informatics, Inc. D.W.B. reports grants and personal fees from EarlySense (Ramat Gan, Israel), personal fees from CDI Negev (Beer-Sheva, Israel), equity from ValeraHealth, equity from Clew (Netanya Israel), equity from MDClone (Beer-Sheva, Israel), personal fees and equity from AESOP (Cambridge, Massachusetts, United States), and grants from IBM Watson Health (New York, New York, United States), outside the submitted work. The remaining authors have no competing interests.

Protection of Human and Animal Subjects

This study was approved by our Institutional Review Board and the requirement for written consent was waived.

• References

• 1 Nanji KC, Patel A, Shaikh S, Seger DL, Bates DW. Evaluation of perioperative medication errors and adverse drug events. Anesthesiology 2016; 124 (01) 25-34
• 2 Merry AF, Webster CS, Hannam J. et al. Multimodal system designed to reduce errors in recording and administration of drugs in anaesthesia: prospective randomised clinical evaluation. BMJ 2011; 343: d5543
• 3 Hodkinson A, Tyler N, Ashcroft DM. et al. Preventable medication harm across health care settings: a systematic review and meta-analysis. BMC Med 2020; 18 (01) 313
• 4 Barker KN, Flynn EA, Pepper GA, Bates DW, Mikeal RL. Medication errors observed in 36 health care facilities. Arch Intern Med 2002; 162 (16) 1897-1903
• 5 Kaushal R, Bates DW, Landrigan C. et al. Medication errors and adverse drug events in pediatric inpatients. JAMA 2001; 285 (16) 2114-2120
• 6 Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies. JAMA 1998; 279 (15) 1200-1205
• 7 Magrabi F, Li SY, Day RO, Coiera E. Errors and electronic prescribing: a controlled laboratory study to examine task complexity and interruption effects. J Am Med Inform Assoc 2010; 17 (05) 575-583
• 8 Devine EB, Hansen RN, Wilson-Norton JL. et al. The impact of computerized provider order entry on medication errors in a multispecialty group practice. J Am Med Inform Assoc 2010; 17 (01) 78-84
• 9 Gandhi TK, Weingart SN, Seger AC. et al. Outpatient prescribing errors and the impact of computerized prescribing. J Gen Intern Med 2005; 20 (09) 837-841
• 10 Nanji KC, Rothschild JM, Salzberg C. et al. Errors associated with outpatient computerized prescribing systems. J Am Med Inform Assoc 2011; 18 (06) 767-773
• 11 Webster CS, Larsson L, Frampton CM. et al. Clinical assessment of a new anaesthetic drug administration system: a prospective, controlled, longitudinal incident monitoring study. Anaesthesia 2010; 65 (05) 490-499
• 12 Gandhi TK, Weingart SN, Borus J. et al. Adverse drug events in ambulatory care. N Engl J Med 2003; 348 (16) 1556-1564
• 13 Pharmacists ASoHS. Standardize 4 Safety. Accessed March 12, 2021 at: https://www.ashp.org/Pharmacy-Practice/Standardize-4-Safety-Initiative
• 14 Eichhorn J. APSF hosts medication safety conference: consensus group defines challenges and opportunities for improved practice. APSF Newsletter 2010; 25: 1-8
• 15 Leotsakos A, Zheng H, Croteau R. et al. Standardization in patient safety: the WHO High 5s project. Int J Qual Health Care 2014; 26 (02) 109-116
• 16 The International Organization for Standardization (ISO). Ergonomics of human-system interaction—Part 210: Human-centred design for interactive systems. Accessed January 16, 2020 at: https://www.iso.org/obp/ui/#iso:std:iso:9241:-210:ed-1:v1:en
• 17 User-Centered Design Basics. Accessed October 25, 2020 at: https://www.usability.gov/what-and-why/user-centered-design.html
• 18 Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc 2015; 22 (06) 1179-1182
• 19 Melnick ER, Dyrbye LN, Sinsky CA. et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc 2020; 95 (03) 476-487
• 20 Bates DW, Leape LL, Cullen DJ. et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998; 280 (15) 1311-1316
• 21 Eslami S, de Keizer NF, Abu-Hanna A. The impact of computerized physician medication order entry in hospitalized patients–a systematic review. Int J Med Inform 2008; 77 (06) 365-376
• 22 Wright A, Feblowitz J, Phansalkar S. et al. Preventability of adverse drug events involving multiple drugs using publicly available clinical decision support tools. Am J Health Syst Pharm 2012; 69 (03) 221-227
• 23 Mishuris RG, Linder JA, Bates DW, Bitton A. Using electronic health record clinical decision support is associated with improved quality of care. Am J Manag Care 2014; 20 (10) e445-e452
• 24 Griffey RT, Lo HG, Burdick E, Keohane C, Bates DW. Guided medication dosing for elderly emergency patients using real-time, computerized decision support. J Am Med Inform Assoc 2012; 19 (01) 86-93
• 25 van Rosse F, Maat B, Rademaker CM, van Vught AJ, Egberts AC, Bollen CW. The effect of computerized physician order entry on medication prescription errors and clinical outcome in pediatric and intensive care: a systematic review. Pediatrics 2009; 123 (04) 1184-1190
• 26 Marasinghe KM. Computerised clinical decision support systems to improve medication safety in long-term care homes: a systematic review. BMJ Open 2015; 5 (05) e006539
• 27 Simpao AF, Tan JM, Lingappan AM, Gálvez JA, Morgan SE, Krall MA. A systematic review of near real-time and point-of-care clinical decision support in anesthesia information management systems. J Clin Monit Comput 2017; 31 (05) 885-894
• 28 Freundlich RE, Ehrenfeld JM. Perioperative information systems: opportunities to improve delivery of care and clinical outcomes in cardiac and vascular surgery. J Cardiothorac Vasc Anesth 2018; 32 (03) 1458-1463
• 29 Freundlich RE, Ehrenfeld JM. Anesthesia information management: clinical decision support. Curr Opin Anaesthesiol 2017; 30 (06) 705-709
• 30 Nair BG, Gabel E, Hofer I, Schwid HA, Cannesson M. Intraoperative clinical decision support for anesthesia: a narrative review of available systems. Anesth Analg 2017; 124 (02) 603-617
• 31 Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2006; 13 (05) 547-556
• 32 Gregory ME, Russo E, Singh H. Electronic health record alert-related workload as a predictor of burnout in primary care providers. Appl Clin Inform 2017; 8 (03) 686-697
• 33 Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform 2012; 45 (06) 1202-1216
• 34 Miller K, Mosby D, Capan M. et al. Interface, information, interaction: a narrative review of design and functional requirements for clinical decision support. J Am Med Inform Assoc 2018; 25 (05) 585-592
• 35 Marcilly R, Ammenwerth E, Roehrer E, Niès J, Beuscart-Zéphir MC. Evidence-based usability design principles for medication alerting systems. BMC Med Inform Decis Mak 2018; 18 (01) 69
• 36 Fitch K, Bernstein SJ, Aguilar MD. et al. The RAND/UCLA appropriateness method user's manual. RAND Corporation. Accessed September 17, 2021 at: https://www.rand.org/content/dam/rand/pubs/monograph_reports/2011/MR1269.pdf
• 37 Mangione-Smith R, DeCristofaro AH, Setodji CM. et al. The quality of ambulatory care delivered to children in the United States. N Engl J Med 2007; 357 (15) 1515-1523
• 38 Hinkelbein J, Lamperti M, Akeson J. et al. European Society of Anaesthesiology and European Board of Anaesthesiology guidelines for procedural sedation and analgesia in adults. Eur J Anaesthesiol 2018; 35 (01) 6-24
• 39 SooHoo NF, Lieberman JR, Farng E, Park S, Jain S, Ko CY. Development of quality of care indicators for patients undergoing total hip or total knee replacement. BMJ Qual Saf 2011; 20 (02) 153-157
• 40 Pagel C, Brown KL, McLeod I. et al. Selection by a panel of clinicians and family representatives of important early morbidities associated with paediatric cardiac surgery suitable for routine monitoring using the nominal group technique and a robust voting process. BMJ Open 2017; 7 (05) e014743
• 41 Utley M, Gallivan S, Mills M, Mason M, Hargraves C. A consensus process for identifying a prioritised list of study questions. Health Care Manage Sci 2007; 10 (01) 105-110
• 42 Nanji KC, Rothschild JM, Boehne JJ, Keohane CA, Ash JS, Poon EG. Unrealized potential and residual consequences of electronic prescribing on pharmacy workflow in the outpatient pharmacy. J Am Med Inform Assoc 2014; 21 (03) 481-486
• 43 Clarke V, Braun V. Successful Qualitative Research: A practical guide for beginners. Thousand Oaks, CA: Sage Publications Inc.; 2013
• 44 O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med 2014; 89 (09) 1245-1251
• 45 International HLS. Fast Healthcare Interoperability Resources. Accessed August 10, 2020 at: http://www.hl7.org/fhir/summary.html
• 46 RedHat. Overview. Accessed January 21, 2020 at: https://developers.redhat.com/products/red-hat-decision-manager/overview
• 47 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 (7494): 765
• 48 Payne TH, Hines LE, Chan RC. et al. Recommendations to improve the usability of drug-drug interaction clinical decision support alerts. J Am Med Inform Assoc 2015; 22 (06) 1243-1250
• 49 Schild S, Sedlmayr B, Schumacher AK, Sedlmayr M, Prokosch HU, St Pierre M. German Cognitive Aid Working Group. A digital cognitive aid for anesthesia to support intraoperative crisis management: results of the user-centered design process. JMIR Mhealth Uhealth 2019; 7 (04) e13226
• 50 Nielsen J. Ten usability heuristics. Accessed September 17, 2021 at: https://pdfs.semanticscholar.org/5f03/b251093aee730ab9772db2e1a8a7eb8522cb.pdf
• 51 Wickens CD, Lee J, Liu YD, Gordon-Becker S. Designing for People. An Introduction to Human Factors Engineering. 2nd ed.. New York, NY: Pearson; 2003
• 52 Merrill D. Book review: Usability Inspection Methods edited by Jakob Nielsen and Robert L. Mack (John Wiley & Sons, Inc., 1994). ACM SIGART Bulletin 1995; 6 (03) 19-20
• 53 Wanderer JP, Ehrenfeld JM. Clinical decision support for perioperative information management systems. Semin Cardiothorac Vasc Anesth 2013; 17 (04) 288-293
• 54 Epstein RH, Dexter F, Patel N. Influencing anesthesia provider behavior using anesthesia information management system data for near real-time alerts and post hoc reports. Anesth Analg 2015; 121 (03) 678-692
• 55 Zachariah M, Phansalkar S, Seidling HM. et al. Development and preliminary evidence for the validity of an instrument assessing implementation of human-factors principles in medication-related decision-support systems–I-MeDeSA. J Am Med Inform Assoc 2011; 18 (Suppl. 01) i62-i72
• 56 Mazur LM, Mosaly PR, Moore C, Marks L. Association of the usability of electronic health records with cognitive workload and performance levels among physicians. JAMA Netw Open 2019; 2 (04) e191709
• 57 Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA 2018; 319 (12) 1276-1278
• 58 Mayhew D. The Usability Engineering Lifecycle: A Practitioner's Handbook for User Interface Design. Burlington MA: Morgan Kaufmann Publishers Inc.; 1999
• 59 National Institute of Standards and Technology.. Schumacher RM, Lowry SZ. NIST guide to the processes approach for improving the usability of electronic health records. Accessed September 17, 2021 at: https://www.nist.gov/system/files/documents/itl/hit/Guide_Final_Publication_Version.pdf
• 60 Middleton B, Bloomrosen M, Dente MA. et al; American Medical Informatics Association. Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA. J Am Med Inform Assoc 2013; 20 (e1): e2-e8
• 61 Johnson CM, Johnson TR, Zhang J. A user-centered framework for redesigning health care interfaces. J Biomed Inform 2005; 38 (01) 75-87
• 62 Katsulis Z, Ergai A, Leung WY. et al. Iterative user centered design for development of a patient-centered fall prevention toolkit. Appl Ergon 2016; 56: 117-126
• 63 Phansalkar S, Edworthy J, Hellier E. et al. A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. J Am Med Inform Assoc 2010; 17 (05) 493-501

Publication History

Received: 15 June 2021

Accepted: 01 September 2021

Article published online:
24 November 2021