Background and Significance
Handoffs, defined as the transfer of patient information, responsibility, and accountability from one clinician to another, are essential for ensuring care continuity during transitions of care.[1] Postoperative handoffs from operating room (OR) to intensive care unit (ICU) settings require an orchestrated coordination of both physical patient transfer in conjunction with transfer of information, responsibility, and accountability between interdisciplinary teams representing anesthesiology, surgery, and critical care.[2] However, they can be vulnerable to communication breakdowns, technical errors, and environmental distractions,[3]
[4]
[5] leading to process failures.[6]
[7]
[8]
Standardization using process-based protocols[9] and structured information transfer checklists[10] are implemented to mitigate these care transition failures. Initial evaluations suggest that these standardized strategies were successful in reducing information loss, technical errors, and process defects while increasing clinician satisfaction and teamwork.[9]
[11] However, based on our recent systematic review, we identified inconsistent evidence on effectiveness of current handoff tools has been inconsistent and mixed[12]
[13] coupled with limited intervention sustainability over time,[14] which can be partially attributed to current tool limitations. Primarily, postoperative handoff tools were (1) lacking support for interdisciplinary teamwork and anticipatory guidance during handoffs, (2) paper based[15]
[16] with few exceptions,[17]
[18]
[19]
[20]
[21] and (3) focused on improving standardization of process-driven protocols[18] with limited support for supporting communication interactions and coordination needs of receiving teams.[22]
To address these limitations, we conducted a user-centered study[23] to explore design requirements for an electronic health record (EHR)-integrated intervention to support effective, efficient, and interactive handoffs supporting interdisciplinary team workflows. We also examined the potential roles and integration of artificial intelligence (AI) and machine learning (ML) via the EHR[24] to augment handoffs that can foster anticipatory management by summarizing scattered EHR elements into concrete risks for the patient.
Methods
Study Setting and Participants
The study was conducted at a large academic medical center with 1,249 staffed beds. Among the 2019 to 2020 discharges, 14,488 surgical patients were transferred from OR to ICU. On-site hospital units included the OR, cardiothoracic ICU (CTICU), surgical ICU (SICU), and neurology–neurosurgical ICU (NNICU). Patients admitted to these ICUs are also remotely monitored by an electronic ICU (eICU), a telemedicine center staffed by ICU clinicians for additional surveillance, and 24/7 support. Participants were recruited with the support of residency and nursing coordinators using a convenience sampling approach.
Existing Postoperative Handoff Protocol
A standardized process-based protocol supported postoperative bedside ICU handoffs.[19] The protocol included (1) process steps to be followed during handoffs and (2) an information transfer report template ([Fig. 1]). Although laminated protocol copies were available at bedside for reference, there were no formal handoff documentation tools. The eICU team observed all handoffs.
Fig. 1 Institutional OR–ICU standardized handoff protocol. ICU, intensive care unit; OR, operating room.
Data Collection
We conducted design workshops in three phases to (1) obtain clinician insights on the current handoff protocol, (2) identify requirements for a handoff intervention with support for communication and documentation, and (3) explore AI integration into our risk assessments to augment postoperative handoff communication ([Fig. 2]). Workshops were audio-recorded and led by C.R.K. (clinician) and J.A. (qualitative expert).
Fig. 2 Phases of design workshops (supported by focus groups).
Phase 1: we used a semistructured guide to discuss the group's perceptions about the current handoff process and gather perspectives on effective OR–ICU handoffs. Participants were oriented to the goal of an integrated handoff intervention and asked to consider how it might fit into their workflow.
Phase 2: we elicited information requirements for an EHR-integrated handoff intervention with closed card sorting. Card sorting[25] was used to explore users' preferences on functionality, overall structure, navigation, and labeling.[26] Participants were given a list of content elements based on prior studies.[1]
[19] Labeled sticky notes were sequentially placed on a board visible to all participants. Through group brainstorming, participants discussed and modified their ranking decisions, adding additional elements through nomination.
Phase 3: we adopted a scenario-based design ideation approach to gather intervention design ideas and elicit feedback on low-fidelity intervention sketches (printed sheets).[27] Handoff scenarios were drawn from our retrospective database.[28]
[29] Additionally, to examine the potential utility of AI- and ML-generated risk assessment during handoffs, we supplied cross-validated risks for adverse events[28]
[29]
[30] (acute kidney injury [AKI], arrhythmia, pneumonia, acute heart failure, delirium, reintubation, unplanned ICU admission, wound infection, and venous thromboembolism). Three scenarios representing a diversity of adverse event risks were selected ([Fig. 3]).
Fig. 3 Scenarios used for design ideation and prototyping (phase 3). AKI, acute kidney injury; CAD, coronary artery disease; EBL, estimated blood loss; VTE, venous thromboembolism. *Substantially elevated risk for adverse events using published methods.[28]
Clinician participants were given scenario narratives, printed deidentified assessments, and anesthesia records. They were also shown the ML-risk predictions used to screen cases in a variety of formats (tabular, graphical, decomposed in force plots,[31] relative and absolute scales, and various reference points). “Important variables” for the prediction were identified using Shapely values[31] and permutation-based importance. Similar questions from phases 1 and 2 were used to gather their perspectives within the context of these scenarios in phase 3.
Data Coding and Analysis
After reading focus group transcripts multiple times, we assigned each statement with data-driven (or open) codes.[32] Similarities and areas of overlap between codes based on relationships were identified to synthesize unifying codes into subthemes. Finally, these subthemes were compared against each other based on similarities to generate higher level themes within and across transcripts. This involved multiple rounds of review and refinement based on theme relevance to our study objectives (see [Supplementary Appendix A] for coding example [available in the online version]). All transcripts were independently coded by authors (J.A. and A.M.), and all discrepancies were discussed to achieve team consensus. Information elements from card sorting were tallied based on both frequency of clinician selection and ranking of importance.
Results
A total of 24 participants (5 clinical anesthesiology fellows, 9 ICU registered nurses, and 10 anesthesiology/critical care residents) participated across four design workshops were conducted. Each workshop lasted an hour on average. We report on major themes identified: factors affecting handoff protocol use and sustainability, intervention components and role of ML-generated risks in handoff intervention, and intervention design requirements and implementation features. Representative quotations and additional data are provided in the [Supplementary Appendix B] (available in the online version).
Factors Affecting Postoperative Handoff Protocol Use and Sustainability
Two major factors impacted handoff protocol use and sustainability. First, over half of the clinicians believed there was a lack of awareness about the current standardized protocol. Without ongoing training, compliance with the standardized protocol was perceived to be limited. Residents especially reported that although some of them knew about the protocol, no one adhered to it: “I've never used it, nobody goes by (the protocol) ….” (Res-4)
Second, the EHR failed to adequately support effective information transfer during handoffs. There was limited awareness on “how to” access and interpret the pre- and intraoperative information found in the anesthesia record by ICU clinicians, “where to” document handoff information by OR clinicians, and “what” information in the record was critical and essential for maintaining care continuity versus irrelevant.
“If you want to look at the actual intra-op and look at what was going on in a concise, easy format, you have to actually look at the intra-procedure tab on the anesthesia thing. Otherwise it's kind of confusing and disorienting.” (Res-1)
Almost all clinicians agreed that anesthesia details were often hard to find but important to include in a handoff intervention.
Handoff Intervention Elements
Core Elements
The importance of elements was determined through card sorting and open-ended interview questions. [Fig. 4] shows how frequently participants chose each element. Elements viewed as necessary by more than half of our clinicians were considered core elements. A full list of participant-selected elements is presented in the [Supplementary Appendix C] (available in the online version).
Fig. 4 Frequency of element selection. Red text indicates free-text suggestions from participants to include in our intervention. BMI, body mass index; OSA, obstructive sleep apnea; EBL, estimated blood loss;
Among participants, residents tended to mark more elements as important to include in the intervention, while fellows and nurses tended to ignore items such as “age” and “preoperative diagnosis.” [Table 1] shows the frequency with which elements were viewed as important by participants. Additional elements included diuretics given (once), paralytic reversal given (four times), and endotrachial tube size and position (once).
Table 1
Prioritization of content elements
|
Element
|
No. of participant votes who viewed these elements as one of the top 10 most important elements to include
|
|
Age, EBL, blood products given, difficult intubation
|
10+ participants
|
|
Preoperative diagnosis, scheduled procedure, duration of anesthesia, crystalloid given, colloid given, antibiotics given (and time), neuromuscular blockade given (twitches and TOF), opioid analgesics given, arterial line present, central line present, preinduction vital signs
|
6–9 participants
|
|
Allergy list, regional block present, high-risk OSA, height/weight/BMI, average vital signs last 15 minutes, β-blockers given/continued, premedication given, insulin given
|
5 or fewer participants
|
Abbreviations: BMI, body mass index; EBL, estimated blood loss; TOF, train of four; OSA, obstructive sleep apnea.
Flexible Elements and Machine Learning–Generated Risk Predictions
Items that could be included depending on patient pre- and intraoperative management and postoperative risks, such as risk of VTE, intraoperative abnormalities, risk of pneumonia, 30-day mortality, and risk of AKI, were regarded as “flexible elements.” Most residents and nurses believed lines, average vital signs within the last 15 minutes, and insulin given were important to include, while fellows did not. In contrast, many critical care fellows found that anesthesia providers tended to focus on intraoperative management details of little meaning to them, stating they only wanted to know intraoperative information if it directly affected their ICU care.
“Why do we care about the dose of fentanyl? Why do we care about the opioid dose? (The anesthesiology team is) like, ‘We gave 600 of fentanyl.’ I'm like, ‘I don't care.’ Why do I care how much you gave intra-op? I don't know. I don't know how that affects me.” (Fellow-2)
More than two-thirds of our clinicians were only interested in actionable or modifiable risks. Information that would not affect care was irrelevant to them, and some clinicians worried that receiving reports on nonimmediate concerns would only increase their workload.
“We get a lot of the global (concerns) from teleICU too. It would just be double the work.” (RN-7)
There was, however, significant interest in the comparison of case patients to the average patient pre-, intra-, and postoperatively. Both nurses and residents stressed that understanding baselines were crucial in interpreting intraoperative data.
“I think having their pre-op info is good because then we know what their baseline is. As far as the averages for everyone else, it gives us an idea of where they should be as opposed to where they are, which is useful, and then what they actually are. So that way we know before surgery their baseline function was this. After surgery it should be this, but theirs is actually this one so we know if something's going well or something didn't go so good.” (RN-4)
Those responsible for interpreting risk information and adjusting patient care plans accordingly only reported interest in significantly elevated risks and use of various thresholds to distinguish risk severity. There was mixed feedback on how awareness of risks would affect patient management over the course of patients' ICU stays. While residents said they could use the risks to develop patient-centered care plans, nurses and fellows believed this risk information would not affect their patient care.
Unnecessary Elements
Elements including allergies, transfusions, and most intraoperative medications were considered unnecessary to include within an EHR-integrated handoff intervention. Nurses believed antibiotics can also be excluded. High-risk obstructive sleep apnea (OSA), height, weight, and body mass index (BMI) were frequently noted by receiving clinicians as never to be included in the intervention but garnered some interest from residents on the sending team.
Most clinicians reported that unnecessary information would lead to information overload and clutter. Fellows were particularly vocal in their beliefs that only crucial or hard to find information should be included in the intervention.
Clinicians had mixed feelings on whether information that was verbally communicated should also be documented on an intervention. Half of our clinicians believed verbal report information should be included while the other half believed it should not.
“I feel like those things … could be verbally communicated to the team that's receiving the patient, and then at 2 hours later, nobody cares. So the fact that this is going to stay in the patient's chart for a week while they're in ICU—I don't care anymore. I feel like a lot of that will clutter up the sheet and make it much harder to get the couple big things you want to see out of it.” (Fellow-2)
Nurses further explained that they heard three different accounts of a patient's status from three different types of clinicians, and felt that confirming information accuracy (and getting the facts straight) was difficult when accounts were not necessarily reliable. Therefore, verbal report information should also be included in an EHR intervention so they could verify the documented information during handoff communication.
Intervention Design Requirements and Implementation Features
Structural Presentation Format and Visualization Considerations
Clinicians suggested that a yes/no format could be implemented to present certain core elements (e.g., airway) across the header of the intervention, similar to how the EHR interface provided information in the past (A.1.10).
“And [information access to certain patient information] it actually used to be easy. And now, since they reformatted, I think it's hard. Because it used to be when it was in the—they had the header at the top, if you clicked on where it said Difficult Airway ‘Yes/No,’ it would actually bring up their most recent intubation document.” (Fellow-1)
When asked about the format for presenting risks, clinicians unanimously preferred absolute risk statements over percentiles and effect sizes.
“I mean in certain things, there's certain criteria that are gonna be elevated in different patients that's gonna make [patients susceptible to] VTE likely. But looking at the specifics on that particular patient that put them over the top that may be helpful.” (RN-2)
All nurses and some residents strongly preferred qualitative risk descriptions over absolute numbers. Residents particularly preferred graphs to visualize risks.
“For me, I would just like the graphs. Everything else would be too much data. But I'm not (other resident's name), so ….” (Res-9)
Irrespective of presentation choice, all clinicians strongly desired explanations of risks (to understand features contributing to predictions) in addition to the absolute score/qualitative narrative. These clinicians felt that knowing which pre- and intraoperative features explained the elevated risks would provide insights for postoperative management.
Intervention Modality and Access Considerations
Clinicians believed that a handoff intervention integrated into the EHR would be more useful than paper. Additionally, residents and fellows thought they would prefer to access an EHR-integrated intervention directly over a phone or computer, as they tended to be more mobile; however, the only concern was the lack of computer access in certain instances. Nurses stated that having a snapshot of the patient handoff within the EHR would “help (them) take care of the patient and anticipate needs …” (RN-1). However, they preferred to print the intervention form for personal use and control (i.e., editing, perusing information) at bedside.
“We know the patient's coming and we can go into documents and their chart and just automatically print it before they come. We would like if we have control of (printing) it.” (RN-4)
Nevertheless, regardless of the modality, all clinicians preferred to access the intervention before bedside handoff to better prepare and used time during the verbal report to ask appropriate questions.
Discussion
As recommended by the Joint Commission,[33] several U.S. hospitals have implemented handoff tools that adhere to structured information transfer and standardized handoff processes to improve safety during care transitions.[11] While these tools improve rate of information transfer, reports suggest limited sustainability in certain process and clinical outcomes over time.[9]
[34] Furthermore, operative details are often prioritized over anticipatory guidance.[22]
[35] As suggested in this study, this might be due to the inclusion and prioritization of some elements in standardized interventions or patient information irrelevant to specific postoperative care. Ascertaining which data elements are relevant to the receiving care team is crucial in preventing information overload and reducing the risk for care transition failures.[36]
Furthermore, in a study conducted on individual clinician performance, standardized lists of risks were seen to drive action in only a few clinicians.[37] Hence, a balance between standardization and adaptive flexibility is necessary to ensure timely and seamless patient care.[38] This is consistent with our findings that point toward communicating individualized, situational topics, such as postoperative risks, that are critical for implicit handoff functions (e.g., anticipatory guidance and contingency planning). These points of communication prepare the receiving team to better manage postoperative complications and anticipate related resource needs.[6]
Adaptive and patient-centered handoff interventions can potentially mitigate some of the standardized protocol compliance issues along with interdisciplinary teamwork gaps. In developing these interventions, we can streamline the handoff process, support transfer of core elements (pre- and intraoperative), highlight flexible elements including ML-generated patient-specific risks, promote a shared understanding about expectations (or “common ground”) among interdisciplinary teams, and, lastly, require minimal clinician effort for handoff preparation with EHR integration.[10]
Furthermore, we emphasize that any adaptive handoff tool is meant to augment rather than replace verbal handoff communication. For example, electronic tools cannot include information which has not been charted (e.g., subjective assessments and rationales). Our study pointed to important pieces of information that may not be documented before handoffs (e.g., extubation details, sedation for transport, or rescue medications immediately before or after extubation). Like any other form of missing data, incomplete charting/documentation can reduce accuracy of risk predictions. However, EHRs include time of handoff documentation, and deep learning techniques can both impute missing data and recognize missing data patterns.[39]
[40]
[41] Automatic identification of “probably missing” data in a handoff tool can potentially remind the sending team to fill-in potential documentation gaps during the verbal exchange.
Data visualizations are commonly used to communicate risks; however, design and presentation of these risks are crucial in influencing risk perception and decision-making.[42] Risk perception and accuracy of participant inference was the greatest in prior studies when icon arrays were present.[43] Additionally, the ability to see disease or risk progression was crucial when considering treatment options during development of clinical risk report intervention prototypes, simple graphs and designs were preferred and complex visualizations were rarely utilized to their full potential.[44] These preferences aligned with our findings, where many clinicians stated that they preferred simple, color-coded graphs for visualizing trends.
Limitations
Our study comes with limitations. First, given the exploratory nature of this work, the study used a small sample size of participants from a single site, with an uneven distribution of clinicians. This mixed cohort and distribution may skew which elements were prioritized. However, the main intent of our user-centered design (UCD), the study was to focus on exploring innovative design ideas and conducting low-fidelity prototype evaluations to ensure the development and implementation of a user friendly intervention that can be easily integrated within the clinical workflow and the EHR system. Prior work on similar UCD methods found that the number of stakeholders typically involved is low, commonly between 6 and 12 users per focus group and 10 to 20 users involved in card sorting.[45] Second, the group dynamics underlying the design focus group workshops varied. While we facilitated discussion among participants, we did observe participants who dominated certain conversations and at times, swayed the opinions of others. We attempted to mitigate this effect through a multimethod approach to ensure individual opinions were collected without discussion with other participants. Third, during our design workshops, physician participants provided their perspectives, both as a sender and a receiver given their clinical practice and experience in both roles. We were hence unable to make concrete distinctions between element preferences in our analysis. To address these limitations, we are recruiting additional participants for a more balanced distribution of both sending and receiving teams including surgery and certified registered nurse anesthetists. Lastly, we acknowledge that clinician overreliance on adaptive postoperative handoff interventions can be prone to information omissions. However, we believe that such interventions should serve as cognitive aids supporting handoffs, similar to clinical decision support systems.
