Keywords
medication safety - clinical decision support - patient safety - user-centered design - user feedback
Background and Significance
Background and Significance
Medication-related incidents, including medication errors (MEs) and adverse medication events (AMEs), are common in the perioperative setting.[1]
[2]
[3] Perioperative ME rates (4–11%)[1]
[2] are consistent with ME rates in other hospital areas, such as inpatient wards (5–19%)[4]
[5]
[6]
[7] and outpatient clinics (7–12%).[8]
[9]
[10] However, 10 to 13 medications are administered per operation,[1]
[11] resulting in a high percent of operations involving medication-related incidents.[1] Almost half of medication-related incidents involve observed patient harm and the remainder have the potential for harm.[1]
[10]
[12] Thus, preventing MEs in the operating room (OR) is of great public health importance and has become a priority locally, nationally,[13]
[14] and internationally.[15]
Medication use in the OR today presents particular patient safety challenges because it often bypasses standard safety checks, such as electronic order entry with decision support and nursing double checks prior to medication administration. In fact, the OR is one of the few locations where every step of the medication use process (medication selection, dispensing, preparation, administration, documentation, and monitoring) is typically completed by a single clinician (the anesthesia clinician), without safety checks by a second clinician or by clinical decision support (CDS) with alerts to warn of MEs.
Two main features of the OR limit the use of existing medication-related electronic CDS. First, there are typically no prospective medication orders in the OR. Documenting medication in the anesthesia information management system (AIMS) functions as both a retrospective order and documentation that the medication was administered. Second, surgical patients are often among the highest acuity patients in the hospital, and due to the nature and potency of medications administered in the OR, patients' conditions can quickly change while under anesthesia.
User-centered system design is an iterative approach that involves users from initial concept development through implementation, with the goal of making systems usable by focusing on user requirements, and by applying human factors/ergonomic principles and usability techniques.[16]
[17] While there is widespread agreement that user-centered design should be used in electronic health record (EHR) development and if it is incorporated the EHR is more likely to be effective, user-centered design principles are often not followed.[18] There have been broad concerns about EHR usability and in a recent study, U.S. physicians gave EHRs an “F” rating on usability.[19]
Certain EHR functionality (such as CDS and alerts) has been incorporated less frequently in the OR, in part because of the pace of care and the nontraditional medication use process. CDS has been shown to reduce MEs in many non-OR locations, including inpatient wards,[20]
[21]
[22] outpatient clinics,[23] emergency rooms,[24] intensive care units,[25] and long-term care facilities.[26] Intraoperative medication-related CDS on the market currently is often limited to simple reminders (e.g., to redose antibiotics, monitor blood glucose, or administer postoperative nausea prophylaxis).[27]
[28]
[29]
Poor CDS design can lead to negative consequences, such as AMEs and increased provider burden.[31]
[32]
[33] While there is a paucity of data on user-centered design of intraoperative CDS applications, the usability of CDS applications has been well studied outside of the OR.[33]
[34]
[35] To maximize the safety benefits of intraoperative CDS as it becomes more robust and incorporates increased functionality, we believe it must be built using user-centered design.
Objectives
The purpose of this paper is to describe the development and analysis of user feedback for a robust, real-time medication-related CDS application for use in the OR. The user feedback will be used together with future summative usability testing to inform the design of the CDS application.
Methods
The study was conducted at a 1,046-bed tertiary care academic medical center that performed approximately 150 operations per day in 58 ORs excluding off-site anesthetizing locations. There were 276 anesthesia clinicians, including 115 anesthesiologists, 56 certified registered nurse anesthetists (CRNAs), and 105 house staff. The anesthesia clinicians used an electronic AIMS (Epic, Verona, Wisconsin, United States) to document patient demographics, vital signs, medications, perioperative events, and related procedures. Each OR had a barcode-assisted syringe labeling device (Codonics, Middleburg Heights, Ohio, United States) that printed color-coded syringe labels at the time of syringe preparation. Prefilled syringes were also available for select medications. This study was approved by our Institutional Review Board and the requirement for written consent was waived.
We developed a perioperative CDS application with a multidisciplinary team, including individuals with expertise in clinical anesthesia, human factors, informatics, statistics, system design, usability, and programming. The CDS application has been implemented in the ORs at our institution. Application development occurred in the following six phases: (1) identification of need, (2) alert algorithm development, (3) determination of technical requirements, (4) analysis of user feedback, (5) system design, and (6) user interface design.
Identification of Need
We conducted a prospective observational study of 277 operations (with >3,600 medication administrations) at our center and found that 5.3% of medication administrations involved an ME and/or AME.[1] We found that more than half of the MEs, nearly all of the AMEs, and one-third of the potential AMEs (near misses) had the potential to be prevented by CDS with alerts.[1]
Alert Algorithm Development
We mapped each ME and/or AME in our observational study[1] to a CDS alert rule that would have prevented it. For example, if an AME involved hypotension, defined as a mean arterial pressure (MAP) <55 mmHg, the corresponding CDS rule was “alert if MAP <55 mmHg.” We then used the extensively studied RAND-UCLA Delphi's Method[36]
[37]
[38]
[39] to achieve consensus among 25 international experts on the most important CDS rules to include. The underlying goals were to (1) minimize the number of alerts, especially hard-stop alerts, to prevent alert fatigue; and (2) include those alerts with the highest potential to prevent serious and life-threatening MEs.
This process resulted in two main categories of alerts: (1) medication-triggered alerts and (2) reminder alerts. Medication-triggered alerts display prior to medication administration if the CDS algorithms detect a ME, such as a life-threatening medication allergy, when the medication syringe is scanned. Reminder alerts are triggered by real-time, intraoperative patient data, including vital signs, laboratory values, and ventilator data. If the CDS algorithms detect a recommended action that has not yet been performed, a reminder alert will display. For example, if the MAP drops below a critical value (that is individualized based on patient comorbidities and baseline blood pressure) for an extended time, an alert will display to remind the clinician to address the blood pressure.
We used a Condorcet ranking method called the crowd ranking kit[40]
[41] to assign alerts to be either interruptive (critically important alerts that pop up over the standard EHR workflow) or integrated into the workflow, such as customized default medication doses, based on the patient's weight, age, and/or renal function.
Determination of Technical Requirements for Perioperative Clinical Decision Support
In consultation with our informatics specialists (K.C.N., A.B., W.J.G., and D.W.B.), we identified three technical requirements for implementation of the perioperative CDS alert algorithms.
Real-Time Functionality
It is especially important in the fast-paced, rapidly changing OR environment that CDS alerts be able to incorporate real-time patient data such as vital signs and medications being administered. For example, CDS should capture whether medications are incompatible with a patient's heart rate in real-time, or whether there may be a drug-drug interaction with a prior medication that was administered seconds ago. Most AIMS do not have access to real-time vital sign data, and instead typically collect these data averaged over 1 minute or more,[30] which is too slow for important CDS functionality in the OR. For example, 1-minute temporal resolution data could lead to alerts falsely firing (e.g., based on a heart rate value from 1 minute ago that has since been corrected). Thus, it is necessary to build the CDS algorithms outside of the AIMS to achieve real-time functionality.
Individualized Alerts
Delivery of anesthesia is often tailored to patient and provider preferences. Anesthesiologists may titrate doses to effect while at the same time considering patients' physiologic limitations such as renal insufficiency, pain, or coronary artery disease, making too much standardization of practice challenging and undesirable. Medication-related CDS alerts in the OR must be tailored to a patient's baseline physiology, incorporating variables such as baseline blood pressure and renal function. This cannot be done with alerts that have static triggers. For example, a blood pressure that is too low for one patient may be acceptable for another. An antibiotic dose for a patient with normal renal function may be too high for a patient with renal insufficiency.
Workflow Change
Anesthesia clinicians typically document medications in the AIMS after administration. For intraoperative CDS to be maximally effective, initiation of medication documentation should instead occur immediately prior to medication administration, allowing an alert to be displayed when necessary to prevent a ME prior to the medication being administered ([Fig. 1]). While this workflow change can be achieved with manual documentation, it may be facilitated by point-of-care barcode scanning of syringe labels immediately prior to medication administration. Point-of-care barcode scanning may also improve efficiency and automate workflow by launching the medication dosing window, so that the clinician simply inputs the dose instead of also manually searching for the medication dosing window. While inputting a predicted dose prior to administration allows for additional CDS functionality, such as wrong dose alerts, it is not always possible to predict final doses when medications are being titrated. Thus, when necessary, final doses can be documented after administration. Also, if multiple medications are being given in a short period of time (e.g., during induction), the workflow should allow for them to be scanned in quick succession, so that the clinician can go back and confirm the doses when time permits. The initial scan of the syringe label barcode (without a final dose) allows for the majority of CDS functionality, such as age-, weight- or renal-based dosing suggestions, and alerts that are independent of dose such as severe medication allergies or drug-drug interactions. The introduction of processes to automate workflow and improve efficiency such as point-of-care barcode scanning, along with strong communication, user feedback, and champions (trusted peer advocates and coaches for change) can facilitate this workflow change.
Fig. 1 Example medication bolus workflow with and without CDS application. AIMS, anesthesia information management system; CDS, clinical decision support.
Analysis of User Feedback
We conducted iterative user-centered design activities including group and individual design feedback sessions with front-line clinician leaders and subject matter experts to gather information about user requirements for the alert content and usability of the CDS application interface. Participants were clinicians who provide anesthesia (including attending anesthesiologists, CRNAs, and house staff), OR pharmacists, nurses, and patient-safety experts.
We began each group and individual design feedback session by providing background on perioperative MEs and describing the CDS application functionality. We then displayed mock-ups of sample alerts that were drawn in Microsoft PowerPoint (Redmond, Washington, United States). An anesthesiologist (K.C.N.) and a user-centered design expert (P.M.G.) developed a semistructured guide (Appendix A), including open-ended questions to prompt discussion about the workflow and clinical usefulness of the alerts, and elicit feedback on the alert content and user interface to facilitate user acceptance. We incorporated feedback until we reached information saturation, the point at which we were no longer gaining new information or feedback from successive design sessions.
Analyses: descriptive statistics are reported as counts with percentages, using Microsoft Excel. Interview notes were analyzed using applied inductive thematic analysis[42]
[43] to develop a list of themes (comments or ideas) to describe the user feedback. Reviewer bias was minimized by having two independent study staff (K.C.N. and P.M.G.) iteratively review the notes and group similar themes together. In parallel with design sessions, the reviewers met weekly to discuss and iteratively modify the themes, identify emerging themes, and further delineate the relationships among them until they reached consensus on all themes. This manuscript adheres to the Standards for Reporting Qualitative Research (SRQR) guidelines.[44]
System Design
To maximize generalizability and to allow for real-time functionality (see “Determination of Technical Requirements for Perioperative Clinical Decision Support” section), we built the CDS application outside of our AIMS. The clinician launches the CDS instantaneously via a button in the patient's chart, and the CDS runs behind the EHR/AIMS (on the same computer monitor), receiving patient and provider context from the EHR. The CDS inputs and outputs are illustrated in [Fig. 2]. The CDS consists of the following three modules:
Fig. 2 CDS application inputs and outputs.
-
Streaming rule engine: the streaming rule engine creates alerts based on real-time data from the vital sign monitor and ventilator.
-
Synchronous rule engine: the synchronous rule engine receives inputs from the EHR and creates alerts when necessary at the start of a case (e.g., missing laboratory tests when indicated) and in response to intraoperative medications administered or laboratory results. The engine extracts patient data from the EHR via the Health Level 7 (HL7) Fast Healthcare Interoperability Resources (FHIR) standards[45] at the start of each case (e.g., patient demographics) and during the case in real-time (e.g., intraoperative laboratory values). The engine also extracts medication data from barcode scans. Using an off-the-shelf barcode scanner, clinicians scan the barcode on any type of syringe label (manufacturer-applied label, pharmacy-applied label, etc.) immediately prior to medication administration. The scan sends the medication's National Drug Code which serves as a universal medication identifier to the rule engine and also launches the CDS medication dosing window.
-
Web application: the web application was developed and refined iteratively based on user feedback. It sits behind the AIMS on the existing anesthesia workstation computer monitor, and displays windows on top of the AIMS as needed to provide two functions: (1) it presents alerts to the user in real-time, and (2) it presents medication dosing windows that are populated with individualized dosing information (e.g., renal-, age- or weight-based, when applicable), so that the provider just confirms the desired dose either before or after administering the medication. While inputting a predicted dose prior to administration allows for additional CDS functionality such as wrong dose alerts, the initial barcode scan (even without inputting a final dose) allows for a majority of the CDS functionality such as dose suggestions and critical alerts that are independent of dose such as severe medication allergies or drug-drug interactions. After the final medication dose is entered, the medication data are sent to the patient's record in the AIMS for automated documentation in real-time, eliminating the need to manually document the medication in the AIMS.
User Interface Design
The CDS application user interface includes medication dosing windows and alert windows. Medication dosing windows display the medication name and have fields for data input by the user, including dose (with patient-specific quick-choice buttons), route, and administration time. Patient-specific default values are included for all fields ([Fig. 3]).
Fig. 3 Medication dosing window: ketorolac.
The CDS application also generates alert windows, when applicable, for critical alerts (e.g., wrong medication or significant medication allergy), and reminders (e.g., to intervene when indicated by patient monitor data or to check glucose when necessary). The initial alert design was generated from requirements gathered during the Delphi process (see “Methods: Alert Algorithm Development” section) and with our usability specialist (P.M.G.) based on best practices for usability of CDS alerts such as using concise and consistent language, providing actionable tools, including relevant contextual information within the alert, and delivery at the time of decision-making.[33]
[47]
[48] An example alert window is shown in [Fig. 4]. Upon receiving an alert, the provider may decide to accept the alert and revise the action that generated the alert or override the alert and continue with the planned action. Alert overrides trigger an accountable justification prompt which requires the user to indicate a reason for override by selecting from a dropdown menu of reasons or entering a new reason. Collecting and analyzing data on provider response to alerts is essential to keeping alerts relevant, useful, and safe. Alerts with high levels of appropriate overrides can be modified or deleted; those with high levels of inappropriate overrides are good targets for provider educational interventions; and those with high levels of acceptance can remain.
Fig. 4 Alert window.
Results: User Feedback Analysis
Results: User Feedback Analysis
We performed two group and eight individual design feedback sessions. In total, there were 35 participants ([Table 1]). We identified 20 feedback themes corresponding to 19 CDS design changes. Examples of the feedback and our resulting design changes are shown in [Table 2]. The first group design feedback session consisted of a 1-hour session with 21 participants who were members of the department's Quality and Safety Improvement Committee. Participants included eight attending anesthesiologists, four anesthesia house staff, two CRNAs, five patient safety experts, one pharmacist, and one OR nurse. Feedback themes included the following: users did not want alerts with hard stops that force them to take a particular action, users wanted a mechanism to override alerts, and users discussed the implications and the best practices for achieving the required workflow change (initiation of the medication documentation process immediately prior to medication administration instead of after medication administration).
Table 1
Participant characteristics
|
Group session 1
|
Group session 2
|
Individual design sessions
|
Total
|
|
Number of participants
|
21
|
6
|
8
|
35
|
|
Female
n (%)
|
13 (61.9)
|
4 (66.7)
|
1 (12.5)
|
18 (51.4)
|
|
Breakdown by specialty
n (%)
|
|
Anesthesiologist
|
8 (38.1)
|
2 (33.3)
|
6 (75.0)
|
16 (45.7)
|
|
House staff
|
4 (19.0)
|
1 (16.7)
|
–
|
5 (14.3)
|
|
Certified registered nurse anesthetist
|
2 (9.5)
|
–
|
–
|
2 (5.7)
|
|
Operating room nurse
|
1 (4.8)
|
1 (16.7)
|
–
|
2 (5.7)
|
|
Pharmacist
|
1 (4.8)
|
1 (16.7)
|
1 (12.5)
|
3 (8.6)
|
|
Patient/medication safety expert
|
5 (2.4)
|
1 (16.7)
|
1 (12.5)
|
7 (20.0)
|
Table 2
Example feedback themes and corresponding system design changes
|
Example feedback theme
|
Design change
|
|
First group session
|
Users dislike being forced to take a particular action
|
Included a “Reject Suggestion” button for all alerts
|
|
There may be valid reasons for overriding an alert
|
Added accountable justification prompt, which requires users to select a reason for rejecting an alert by choosing from a dropdown menu of acceptable reasons, or entering a new reason as free text
|
|
Workflow change (required to document medications immediately prior to administration instead of immediately after administration) might be challenging to achieve
|
Incorporated point-of-care barcode scanning of syringe label to ease documentation workflow
|
|
Second group session
|
Users may want to change medication units
|
Made unit field editable
|
|
Users may want to edit time of medication administration
|
Made time field editable
|
|
With increased use of sugammadex for reversal of neuromuscular blockade, alerts about potential prolonged response to neuromuscular blockers may be less relevant
|
Deleted alert about prolonged response to neuromuscular blockade in patients with renal failure
|
|
Individual design sessions
|
Users may want even easier access to the alert rationale
|
In the reference link, provided a summary of the relevant text from the reference, in addition to providing the reference citation
|
The second group design feedback session consisted of a 90-minute session with six participants including one OR pharmacist, two anesthesiologists, one OR nurse, one anesthesia resident, and one usability expert. Feedback themes included the following: users wanted the ability to edit values such as the time, units, and computed baseline values (e.g., baseline blood pressure); and users wanted to know the underlying logic and rationale behind alerts. For example, an alert that simply states that “MAP is too low” may be viewed differently if presented along with the rationale “Patient has preexisting hypertension with a baseline MAP of 150 mm Hg.”
Participants in the individual design sessions included six anesthesiologist front-line clinician leaders and two subject matter experts (one pharmacist and one medication safety expert). Their feedback themes included the following: users preferred to receive more information with each alert, and users wanted hard stops alerts only when necessary, and no duplicate alerts.
Discussion
We incorporated a user-centered design process to build a perioperative medication-related CDS application that uses real-time patient data to generate individualized alerts, when applicable. Through group and individual design sessions, we identified 20 feedback themes, corresponding to 19 system changes. Our hope is that incorporating these changes will improve the usability of the CDS application, leading to greater user acceptance. The main themes that we identified for user acceptance were using hard stops only when necessary, providing as much information as feasible about the rationale behind alerts and the clinical context, and allowing users to edit fields such as units, time, and baseline values. Some themes are easily addressed. Providing information about the alert rationale is straightforward and aligns with the important goals of achieving data integrity and transparency. Allowing users to edit fields is also straightforward, though it is essential to clearly indicate the units for a field that will be used in a calculation such as weight. While hard stops should be used extremely infrequently, they can be necessary, for example, to avoid giving a potentially lethal dose of potassium.
While there is a paucity of data on user-centered design of intraoperative CDS applications, Schild and colleagues describe the importance of a user centered design process for a digital cognitive aid that provides checklists for intraoperative crises.[49] The usability of CDS applications has been well studied outside of the OR, with similar findings to ours, including recommendations to design alternatives to hard stops, provide the rationale behind alerts, and allow modifications to orders.[33]
[34]
[35] Horsky and colleagues conducted a targeted review of the best practices for medication-related CDS outside the OR and found that user-centered, iterative design is critical, and recommendations included tiering alerts by severity to avoid excessive use of interruptive alerts, including relevant patient information on dosing screens and using concise text with justifications.[33] The themes we identified also align closely with Nielsen's Usability Heuristics, which are principles that are rooted in human factors engineering and have been extensively used to evaluate the usability of software and web applications.[50]
[51]
[52] For example, our users' preferences for editing fields and avoiding hard stops align with Nielsen's heuristic of User Control and Freedom. Our users' feedback about workflow change aligns with Nielsen's Heuristic of Flexibility and Efficiency of Use, which is an especially important heuristic in perioperative settings where tools with poor usability may take focus away from the patient.
Real-time medication-related CDS in the OR has been difficult to achieve to date.[27]
[30]
[53]
[54] This is one of the first applications of real-time, individualized CDS for medication administration in the OR,[29]
[30] which will likely improve patient safety not only by preventing MEs before they reach the patient but also by reducing the harm of nonpreventable AMEs through alerts to limit the duration of serious hemodynamic abnormalities. While not yet evaluated in the OR, CDS has been shown to prevent MEs in inpatient and outpatient settings, when it has been designed well and human factors issues have been addressed.[20]
[55]
Medication-related CDS in the OR involves a change in the current medication use workflow and any such change must be addressed carefully. Specifically, medication documentation must be initiated immediately prior to medication administration, rather than after administration. By scanning a medication syringe prior to administration, the following two important outcomes can be achieved: (1) automation of the documentation process by scanning instead of manually searching for medications, and (2) allowing CDS functionality to warn of serious MEs before they reach the patient. Given this workflow change, the design of the alerts, their integration into the workflow, and their value to clinicians in terms of reducing cognitive load and improving efficiency are critical to achieving user satisfaction. Changes we made to the initial CDS design based on user feedback were not extensive but they may still have a substantial effect on clinicians' overall willingness to accept the alerts. Alert fatigue represents a high priority concern, and in a critical setting such as the OR (where the “noise” today is already high), clinician feedback should be incorporated into alert design to limit extraneous information and provide actionable tools. Poorly defined user requirements early in software development have been identified as a major contributor to project failure.[56]
[57]
[58] Health care has been slow to adopt a user-centered design approach, but efforts in the last decade to focus on usability have led to a greater understanding of its importance.[16]
[59]
[60] A greater emphasis on early user involvement to elicit user requirements and preferences prior to development can result in time and money efficiencies, as well as a safer and more usable system.[18]
[61]
[62]
Limitations
This study has limitations. First, our participants are members of the anesthesia department at a large tertiary care academic medical center. Their preferences and feedback may not reflect clinicians who work at smaller centers or private practice. Second, the study site is an advanced OR environment with an electronic AIMS and barcode-assisted medication labeling device, which may not be available at other hospitals. To facilitate its generalizability, we designed the CDS to recognize the barcode on any syringe label (manufacturer-applied, OR pharmacy-applied, or barcode-assisted syringe labeling device). The CDS also allows for manual medication entry via a search function (similar to standard AIMS) instead of barcode scanning. Finally, while the CDS application is built on an open-source platform using FHIR international data exchange standards to maximize generalizability, some local implementation and testing of the interfaces may be required at other institutions, as is the case with any clinical application.
Conclusion
We incorporated a user-centered design process to build a perioperative medication-related CDS application that uses real-time patient data to generate individualized alerts, when applicable. Through group and individual design sessions, we identified the following three main requirements for user acceptance: (1) using hard stops only when necessary; (2) providing as much information as feasible about the rationale behind alerts and the clinical context; and (3) allowing users to edit fields such as units, time, and baseline values. Our use of a user-centered design process is likely to improve the usability of the novel CDS application, leading to greater user acceptance. Future work will focus on summative usability testing of the CDS application in a clinical setting, followed by measuring the impact of the CDS application on perioperative ME and AME rates.
Clinical Relevance Statement
Clinical Relevance Statement
Meaningful use of perioperative clinical decision support (CDS) has the potential to reduce the incidence of perioperative MEs[1] including dosing errors, errors of omission, monitoring errors, and wrong medication errors. In fact, our prior work has demonstrated that perioperative CDS with alerts has the potential to eliminate more than half of MEs, nearly all AMEs and one-third of potential AMEs (near misses).[1] The incorporation of user-centered design is critical to achieve user acceptance of perioperative medication-related CDS in general,[63] and we suspect this will also hold true in the perioperative setting, facilitating a reduction in the incidence of MEs, and making surgery and anesthesia safer for patients.
Multiple Choice Questions
Multiple Choice Questions
-
Based on the user feedback presented in this study, the authors determined which of the following features of clinical decision support (CDS) systems are important:
Correct Answer: The correct answer is option c. Choice a is incorrect because user feedback indicated that users did not want CDS to contain hard stops that force them to take a particular action. Hard stops may be necessary at times but should generally be avoided. Choice b is incorrect because the authors found users wanted to be able to edit the time fields, as well as units and baseline values. Answer choice d is incorrect because users did not want duplicate alerts, which can lead to alert fatigue. Answer choice c is correct because, alongside having a reference link, users indicated that they wanted to know the alert rationale.
-
Which of the following did the authors identify as a technical requirement of clinical decision support (CDS) systems in the operating room (OR)?
-
Real-time functionality
-
1-minute temporal resolution data
-
Alerts that are the same for every patient
-
A similar workflow to existing AIMS
Correct Answer: The correct answer is option a. Choice a is correct because in the fast-paced, rapidly changing OR environment, CDS alerts should incorporate real-time patient data. Choice b is incorrect because one-minute temporal resolution data could lead to alerts falsely firing. Choice c is incorrect because CDS alerts in the OR should be tailored to a patient's baseline physiology, incorporating variables such as baseline blood pressure and renal function. Choice d is incorrect because anesthesia clinicians typically document medications in the AIMS after administration. For the CDS to be most effective, clinicians should initiate documentation of medications before administration, so that alerts can prevent medication errors before they reach the patient.
Group and Individual Design Session Guide
Part One: Introductions (2 Minutes)
-
Brief study description: we are designing a clinical decision support application that provides alerts to prevent medication errors in the operating room.
-
Goal of session: we are interested in your feedback today on the content, layout and design of the decision support to learn how we can make it the most helpful.
-
Rules: be open and honest. All feedback is good feedback.
Part Two: Paper Prototype Review (30–40 Minutes)
Discussion of workflow”
-
What do you like about this workflow?
-
What concerns you about this workflow?
-
How would this be most useful to you?
-
Introduction to alert structure.
Medication-triggered alerts
-
What do you like about these alerts?
-
What don't you like about these alerts?
-
How would you expect the “reject suggestion” button to behave?
Renal dosing suggestions: describe to us what information this is conveying to you? What else would be helpful for this screen?
Reminder alerts—vitals
-
What do you like about these alerts?
-
What don't you like about these alerts?
-
What is your preferred level of detail for these alerts (e.g., specific medication suggestions or generic suggestions)?
Reminder alerts—labs
-
What do you like about these alerts?
-
What don't you like about these alerts?
-
What do you think about the option to view references and guidelines?
Part Three: Wrap-up (3 Minutes)
