Behavioral Health Decision Support Systems and User Interface Design in the Emergency Department

Abstract

Objective The objective of this qualitative study is to gauge physician sentiment about an emergency department (ED) clinical decision support (CDS) system implemented in multiple adult EDs within a university hospital system. This CDS system focuses on predicting patients' likelihood of ED recidivism and/or adverse opioid-related events.

Methods The study was conducted among adult emergency physicians working in three EDs of a single academic health system in Rhode Island. Qualitative, semistructured interviews were conducted with ED physicians. Interviews assessed physicians' prior experience with predictive analytics, thoughts on the alert's placement, design, and content, the alert's overall impact, and potential areas for improvement. Responses were aggregated and common themes identified.

Results Twenty-three interviews were conducted (11 preimplementation and 12 postimplementation). Themes were identified regarding each physician familiarity with predictive analytics, alert rollout, alert appearance and content, and on alert sentiments. Most physicians viewed these alerts as a neutral or positive EHR addition, with responses ranging from neutral to positive. The alert placement was noted to be largely intuitive and nonintrusive. The design of the alert was generally viewed positively. The alert's content was believed to be accurate, although the decision to respond to the alert's call-to-action was physician dependent. Those who tended to ignore the alert did so for a few reasons, including already knowing the information the alert contains, the alert offering information that is not relevant to this particular patient, and the alert not containing enough information to be useful.

Conclusion Ultimately, this alert appears to have a marginally positive effect on ED physician workflow. At its most beneficial, the alert reminded physicians to deeply consider the care provided to high-risk populations and to potentially adjust their care and referrals. At its least beneficial, the alert did not affect physician decision-making but was not intrusive to the point of negatively impacting workflow.

Keywords

decision support systems - clinical - emergency department - hospital - informatics - attitude - interview

Background and Significance

Emergency medicine clinicians deliver care in a fast-paced environment, often for patients they have never met before. To reduce cognitive load and deliver the highest-quality care, they need to rapidly integrate information from multiple data sources. Clinical decision support (CDS) systems have emerged as a potential means of streamlining, standardizing, and improving care quality.[1] CDS systems are software products built to directly help clinical decision-making, as patient characteristics are compared with a computerized knowledge base containing other patients' characteristics and encounter data.[1] CDS systems can be deployed within electronic health records (EHRs) to forecast a variety of health outcomes.[2] [3] [4] [5] [6] [7] [8] [9]

Multiple health care settings leverage predictive analytics,[10] [11] [12] defined as applying modeling techniques to forecast outcomes based on historical data. Owing to the clinical complexity of patients in the emergency department (ED), clinicians may not always have time to address patient social health and care coordination needs.[13] Thus, there may be a role for CDS tools in the initiation of behavioral health triage and care management. Opioid overdose is one example of a preventable health outcome that could be predicted and, potentially, avoided using CDS. Individuals seen in an ED for a nonfatal opioid overdose are at significantly higher risk of ED readmission, repeat overdose, and overdose death.[14] In the year following an ED visit for nonfatal opioid overdose, more than one in twenty patients will die.[15] Being able to identify patients at risk for overdose and ED recidivism could help reduce opioid-related morbidity and mortality.[14]

Unfortunately, several roadblocks may hinder the adoption of CDS tools.[11] Poorly tuned algorithms can entrench bias.[16] [17] [18] Further, CDS alerts may be intrusive and excessive, contributing to alert fatigue[19] and potential alert neglect. Additionally, physicians with lower technological literacy may struggle to incorporate CDS systems into their clinical workflow.[20] Speaking directly with physicians about their thoughts regarding an existing alert system may provide insight into CDS tools created for their use, and such research remains relatively under-studied.[21] [22] [23]

In 2019, we deployed a predictive CDS alert system, Emergency Department Smart Notifications (EDSN). EDSN aims to predict the risk of (1) high ED use and (2) future opioid overdose. As of 2018, the nationwide readmission rate was 14%,[24] and opioid overdose rates increased by 29% (to 0.32 ED visits per 100) between 2018 and 2020.[25] Prior to EDSN's official launch, we interviewed physicians to assess perspectives on CDS for behavioral health risk prediction at large. Based partially on their feedback, EDSN was developed and implemented in early 2019. Over the following year, we conducted additional physician interviews to gauge their alert understanding and opinions on its implementation. While these types of CDS systems exist in other health systems, this research helps illustrate the important role of providers throughout technological development. It is crucial for CDS systems to incorporate physician input throughout development and iteration, and this research highlights the results of such efforts.[26]

Objectives

The objective of this qualitative study is to gauge physician sentiment concerning an ED CDS system recently installed in multiple adult EDs within a university hospital system.

Methods

Setting

This study was conducted in three academic medical center EDs within a single health system in May 2019. The EDs enrolled in the study are high-volume sites, with over 220,000 visits in 2020. Of these, 769 were visits for opioid overdose, approximately half of the state total. The 7- and 30-day ED recidivism rates in 2021 are 8.5 and 17.6%, respectively.

Intervention

EDSN is an EHR alert developed by a collaborative of clinicians, researchers, and informaticists at the health system and partner institutions to alert emergency physicians that their patient was either (1) recently treated in the ED and/or (2) is at high risk for opioid overdose. The system leverages a patient's ED medical record and data from Rhode Island Quality Institute (RIQI) in its classification algorithm. RIQI manages the state's Health Information Exchange, operationalizing data from multiple sources to better inform patient care. EDSN generates a risk score estimating potential ED recidivism and/or adverse opioid-related events. The risk score is translated to recommended actions for the provider, including further review of a patient's medical history and extra attention to ongoing care.

Early EDSN ideation began in 2018. After development and design, EDSN was tested behind the scenes to best evaluate model accuracy and gauge provider feedback. EDSN was then implemented into Epic EHRs across all study EDs in May 2019, visible to providers. [Fig. 1] depicts how the alert renders. The alert can also be seen within the “triage,” “workups,” and “reports” windows.

Fig. 1 Image of EDSN alert within the EHR. Inset shows what happens when banner is clicked for the alert for high ED utilization (data presented in the figure are imaginary).

Interviews

We conducted semistructured interviews with emergency physicians working at any of the three study sites. We used a convenience sample, recruiting participants through announcements at faculty meetings, faculty-wide email requests, and posters placed in the EDs and physicians' administrative offices. The first round of interviews was conducted from 2018 to early 2019 by a coauthor trained in qualitative interviewing. The second round of semistructured interviews was conducted from August 2019 to August 2020 by another trained coauthor. Interviews completed before March 2020 occurred in-person, and those after March 2020 occurred via videoconference. Respondents were briefed on the project and these analyses prior to each discussion. Interviews lasted between 30 and 45 minutes and each respondent received a $20 Amazon gift code upon interview completion.

Interviews before EDSN implementation focused on the identification of factors to promote physician buy-in for CDS tools in the ED. Then, visuals of potential alert design, content, and placement were shown to obtain physician input. Clinicians were able to select which designs they found most effective. Interviews after EDSN implementation were more structured and comprehensive. They focused on (1) physicians' prior understanding of predictive analytics and awareness of EDSN implementation; (2) acceptability and clinical applications of the model-generated alerts; (3) feedback on the placement, design, and content of the alerts; and (4) overall alert user experience and suggestions for other applications (including using symptoms to predict COVID-19 infection during interviews occurring after March 2020). The postimplementation interview guide can be found in the [Supplementary Material S1] (available in the online version).

Based on preimplementation interviews reaching thematic saturation at 11 observations, the goal was to match that postimplementation. Coding sequentially, thematic saturation was determined once we observed no emergent subthemes in three consecutive interviews. Interviews were transcribed verbatim and deidentified prior to analysis. This study was reviewed and approved by the Rhode Island Hospital Institutional Review Board.

Analysis

Transcripts were entered into NVivo 12 (QSR International, Burlington, MA, United States). Researchers conducted an inductive thematic analysis, which allows for feasible comparisons with existing research concerning clinician's CDS evaluation.[27] [28] This method was chosen due to the interviews' semistructured nature, and the ways responses could be grouped into themes and subthemes. Coding was performed by either two or three individuals depending on researcher availability. All codes were agreed upon by consensus.

After coding each transcript, codes were grouped into identifiable themes. All team members reviewed each theme and agreed on the final theme and subtheme schema by consensus. We then created a table summarizing each theme and subtheme based on physician quotes ([Table 1]).

Table 1 Physician remarks (postimplementation)

Theme 1: Physicians have variable experience and attitudes about predictive analytics
“I don't have a lot of background. I was intrigued by this because I see the banners. They don't always tell me useful information but sometimes they do”
“I mean I understand the intention but I don't know about their effectiveness. I understand the intention and I understand the purpose”
“They serve an important purpose from a data standpoint about population health-related efforts”
“They are quick tools that are designed to pop up at times when you are making clinical decisions so you might be at a branching point and it helps choose which path to go down”
“I would say unfortunately my general impression is more of being a distraction often and I would say that they become so frequent that you tend to just click them away without noticing them”
“Sometimes they're helpful but sometimes they pop up and I don't understand why they're there. It's not really changing my management, and there are other times where I'll keep it in mind”
Theme 2: Nonclinical education about alerts and education on shift are effective awareness-building strategies
Subtheme 2a: First notification of the EDSN alerts in a nonclinical environment by a physician champion (an email, a faculty meeting, or from a personal interaction)
“I certainly heard [study PI] talking about it. I wasn't involved in the design of it but she just kept saying “we're going to be launching this, we've been working on this with RIQI for a while””
“I think I heard [study PI] talk about it and then it popped up sometime, but I thought it was going to pop up in another manner, but what it is is like a banner”
“Probably came from [study PI] in an email, or attached to some faculty news”
Subtheme 2b: First notification of EDSN alerts was during a clinical shift
“I guess for me I don't know exactly when they started but for me they've always already been there since I started in October”
“I feel like I usually see it when I click on the chart review and see the orange or yellow banner across the top, which I'll then try to click on. We did like an orientation with IT and I think they alluded to them but I don't think we ever had a faculty introduction on the different ones”
Theme 3: EDSN placement, design, and content acceptable but may be optimized by design changes
Subtheme 3a: Placement considered prominent, easy to find, but not obtrusive
“I think [the study PI has] a prime piece of real estate”
“I do like the placement, and that it is available in multiple different tabs and screens depending on what your workflow is because I know everyone uses epic a little bit differently”
“I like that they aren't obnoxious in the sense that they aren't in the way of my workflow, I don't have to click seven buttons to make them go away”
“Thinking about my workflow, sometimes if it's a really busy shift it's hard to keep full track of all of these screens. But it may be useful to get one last reminder to stop and think about this again before I decide to send the patient home”
Subtheme 3b: Physicians appreciate the design but believe colors can be used to better convey messaging
“This is a passive thing that I hope others are using and clicking on but you aren't forced to waste another few clicks for no added value”
“It could be helpful distinguishing the opiate, at risk, from the super utilizer population with a different color”
“I do like the color of the banner that is really helpful. One thought is when you open the notification, I think that page is a bit too black-and-white. Some color or visual aids in that layout could be incorporated that would make it a bit easier to get what you want”
“Color-coding the table based on recency could be helpful. If they are above a certain number that would mean that high utilization has been happening, and knowing that in particular could be helpful”
Subtheme 3c: Notification content viewed as clear, but lacks comprehensiveness
“I think the header makes it pretty clear, the rest of the message can be less clear”
“The content makes sense. If anything it makes me go into the chart and look at what's going on”
“I struggle as a physician to figure out what to do with information and not let it bias me”
“The one on substance abuse is very short, which is good, but it doesn't always tell me what the issue was. It says they screened positive, but screened positive for what? Then I have to go find [the answer]”
“It may be worth highlighting the call-to-action if that's something you guys are interested in”
Theme 4: Physicians had positive attitudes toward potential of EDSN while skeptical of its effectiveness in changing physician behavior
Subtheme 4a: Physicians tended to ignore the alert more often than not, although small changes were noted
“About 90% of the time I ignore it”
“I do ignore things a lot more because there's so many of them, It is useful, I do use the information. I'm sure I could use it more, but it is useful. I used to be anti-banner, but maybe I'm coming back around”
“Over time I have worked to ignore them a bit more than I should or I don't know, it's probably a bit of learned neglect”
Subtheme 4b: Many physicians doubted alert's information would change their clinical decision making
“Sometimes it may be irrelevant. The patient is there frequently with abdominal pain and cyclic vomiting but now they're here because they have a broken arm. So clearly then I don't [click]”
“If someone comes in who may have fell, and had a broken wrist, it's pretty straightforward. Unless they have some other sort of red flag I might not click on it”
“What I'm thinking is “this isn't going to change anything.” Just because they were seen at Kent a few days ago doesn't mean it changes anything”
“I see it, and I feel like nothing it will tell me will change their care anyways, that is why I would ignore it. If the 80 year old comes in and the high utilizer flags, it's like “great they've been here before.” But if they have like 12 comorbidities then it's not surprising. That alert there would not change my ultimate management”
Subtheme 4c: Time limitations influenced decisions to view EDSN alert
“It's going to depend on how rushed I am. If I am really behind and am trying to catch up I might not click”
“Sometimes if it's a really busy shift it's hard to keep full track of all of these screens. But it may be useful to get one last reminder to stop and think about this again before I decide to send the patient home”
Subtheme 4d: A minority of physicians mentioned that the alert affected workflow, but those who said it had an effect recalled it being a positive one
“Maybe if I looked over their care briefly and saw they had labs done but didn't notice they were actually seen at that emergency department, this has clued me into asking more questions about the patient's history because the patients are wishy washy on what happened when, where they have been, sometimes they hide that they've been in other emergency departments because they want you to look at it from a fresh perspective”
“Maybe folks who have a heavier clinical load may have changed their practice patterns a bit more. I can't say I have changed my practice patterns, that I look for these and x, y, and z, but I would say marginally they have had an impact”
Subtheme 4e: Physicians view EDSN as having potential that has yet to be realized
“Some sort of education along the lines of what you use that for would be helpful I mean, you get the notification and the information but if there's no clinical utility or action it just becomes extra information that is easy to ignore”
“I will confess that at the beginning, I wasn't using it very much. And then when I was asked to give a quote about the value of the program and I was like, “before I give a quote I should probably start using it.” And then I used and thought, hey this is pretty good. I hadn't really been clicking on it all that often. My comments all reflect my use in the last 3 months. When it first came out I probably wasn't clicking on it very often but I decided to make a more intentional effort to do so”

Results

We conducted 11 interviews prior to EDSN implementation and 12 interviews afterward. Four physicians were interviewed in both sessions. Researchers sought to include more physicians in both sessions but were primarily restricted by scheduling constraints. [Table 2] displays a summary of respondents' demographic characteristics. The sample was mostly white, male, and skewed slightly younger than the general population of ED clinicians at the health system.

Table 2 Summary characteristics

Characteristic	Preimplementation participants (N = 11)	Postimplementation participants (N = 12)
Age (y)
20–29	1 (9%)	2 (17%)
30–39	6 (55%)	5 (42%)
40–49	1 (9%)	2 (17%)
50–59	3 (27%)	2 (17%)
60+	0	1 (8%)
Gender
Male	7 (64%)	9 (75%)
Female	4 (36%)	3 (25%)
Race
White	9 (82%)	10 (83%)
Black	1 (9%)	1 (8%)
Other	1 (9%)	1 (8%)
Work % spent on various activities
Patient care	48%	43%
Supervising other practitioners	21%	19%
Research	5%	7%
Administrative tasks	21%	26%
Continued medical education	4%	4%
Other	1%	1%

Preimplementation

Most interviewees were familiar or had personally interacted with CDS technology prior to EDSN implementation. Concerns included lack of trustworthiness, loss of physician agency, legal liability, devaluing the “art of medicine,” and reimbursement.

Participants expressed discordant preferences about notification design (e.g., location, color, and content). Nearly all physicians perceived CDS as potentially helpful for overdose prevention, while only roughly half perceived utility for reducing ED recidivism. Indeed, physicians generally expressed that information on frequent ED visitation was not easily actionable. With overdose risk, on the contrary, physicians believed that EDSN notifications might readily impact their opioid and naloxone prescribing practices. In totality, providers preferred alerts that automated documentation, synthesized established risk factors, and contained actionable recommendations. Preferences did not vary according to age and experience level.

Regarding trustworthiness, physicians suggested that decision maker's experience plays a key role. One respondent supported this in saying, “I think I'd say I trust [CDS] more than I trust a second-year med student to [interpret risk]. But do I trust it more than someone who's been practicing for 60 years? … it depends” (‘Before’ Participant [BP] 2). Behind these worries in trustworthiness is the belief that CDS systems may strip physicians' ability to act on individual judgment. One physician stated, “[CDS] becomes a reoccurring barrier to getting the job done, getting the work done, being able to complete the patient care and moving onto the next patient. So it basically becomes a frustrating agent without adding any real perceived value to their care” (BP 5).

An extension of this concern was the worry that acting on these alerts may provide evidence for legal or reimbursement concerns. Those who brought up this concern were strongly against existing CDS alerts being used for these nonclinical purposes. These concerns collectively implied a worry that the subjective interpretation integral to medical expertise will be sacrificed for rigid algorithms. This sentiment is articulated by one physician who stated, “There are actually a lot of clinical decision rules that say if “x” then “y” if “y” then “z,” but at the end of it all your gestalt actually trumps all of the rules and science and data” (BP 6).

Other potential barriers to behavioral health use CDS discussed included ED physician time limitations, lack of effective treatment options, and difficulty facilitating linkages to outpatient services. Physicians suggested that a successful iteration of EDSN would address these concerns while remaining a nonintrusive, supplemental tool.

Postimplementation

We identified four overarching postimplementation themes regarding how physicians perceive CDS systems in general and EDSN alerts specifically ([Table 1]).

• Theme 1: Physicians have variable experience with and attitudes toward predictive analytics: few respondents had clinical experience with predictive analytics, with only one having previously helped implement a CDS algorithm. Nonetheless, respondents felt they understood the general idea and intended purpose.

About half of the respondents noted the positive aspects of predictive analytics. They mentioned that clinical informatics can help generate alerts that serve as assistants to clinicians' daily workflows. Physicians suggested that alerts might optimize physician decision-making processes by synthesizing overlooked or convoluted information.

A few respondents explained potential drawbacks of predictive analytics-derived CDS systems. The frustration with seeing too many alerts appeared to be the primary issue. One provider echoed this when stating, “I would say unfortunately my general impression is more of being a distraction often and I would say that they become so frequent that you tend to just click them away without noticing them” (‘After’ Participant [AP] 5). Additionally, some noted that they paid more attention to these alerts as trainees but tuned them out increasingly as time progressed. Ultimately, physicians believed that low user uptake made it difficult to tell if predictive analytics has had an impact at the hospital level.

• Theme 2: Nonclinical education about alerts and education on shift are effective awareness-building strategies: some participants were made aware of the EDSN alert via email or faculty meeting while others reported seeing the alert for the first time during a shift.

Those who mentioned they recall learning of EDSN before seeing it live said this occurred through an email, a faculty meeting, or from a personal interaction with the study principal investigator. The variety of ways physicians learned of these alerts could have contributed to differing levels of alert understanding and familiarity.

Of the respondents who remember their awareness of the EDSN alerts occurring during a shift, a couple mentioned that they may have had prior knowledge of the alert but had forgotten about it by the time they saw it live. Other doctors, especially those with fewer years of experience, mentioned seeing these alerts for the first time during a shift because they had only recently begun working at the hospital.

• Theme 3: Providers deemed EDSN placement, design, and content to be acceptable but may be optimized by design changes: in terms of alert frequency, most physicians recalled seeing EDSN flags accompany 10 to 25% of patients' charts. Most respondents believed that this frequency was consistent with what they would expect. Further, algorithmic bias was not seen as a serious concern by most clinicians, while acknowledging that this does not mean bias is not present.

Physicians generally agreed that the alert had useful placement and appreciated its ability to render in different tabs and screens. As one clinician stated, “I do like the placement, and that it is available in multiple different tabs and screens depending on what your workflow is because I know everyone uses Epic a little bit differently” (AP 12). Multiple physicians believed EDSN is less intrusive than other EHR alerts. Physicians suggested that these alerts could also render within chart review, at discharge, color coded within the patient list, as a clinical note, and within the “ED activities” tab.

Regarding EDSN design, most physicians appreciated that the alert does not “pop up” without prompting and that these alerts do not render without being clicked. Multiple people also suggested that color coding based on the alert content could help physicians quickly deduce the patient's risk of ED recidivism or the number of recent ED visits without having to investigate further. Emojis were also suggested to quickly preview the information enclosed in the alert.

Most respondents stated that the wording and substance of the alerts are clear; however, they could be more comprehensive. While its brevity is appreciated, clinicians said that the alert does not fully explain the nature and severity of the risk, requiring physicians to spend extra time tracking down the required information.

While physicians supported the “suggested next steps” function, they wished the call-to-action could be more explicit. Physicians suggested that this could display as a button directing physicians to order additional tests or viewing previous visits in greater detail.

• Theme 4: Physicians had positive attitudes toward the potential of EDSN while skeptical of its effectiveness in changing physician behavior: while most physicians believed the alert to be sufficiently calibrated, most respondents—especially more experienced physicians—ignored the alert more frequently than not. These physicians noted that the steps prompted by the alert described their normal courses of action. Even if the alert did not change their clinical processes, some still acknowledged its helpfulness by confirming their decisions.

Clinicians commonly ignored the alert because it did not alter their clinical process. This could be the case for several reasons. First, multiple ED visits could be independent of each other, rather than a pattern of risky behaviors. Additionally, if the physician has previously engaged with this patient, the alert may not provide new information. Another reason to ignore the alert involved the cumulative “alert fatigue” that physicians face.

Further, some respondents mentioned that individual bandwidth may influence responsiveness to EDSN alerts. These physicians noted that they are inclined to ignore the notification if they were running behind schedule, contending with other significant clinical demands. Others, however, mentioned how the alert served as a useful care reminder during busy clinical shifts. Physicians stated that this aids in preventing excessive procedures, especially among patients who may not be forthcoming about their medical history. Similarly, physicians appreciated how it worked across hospitals and could include external information. A couple of physicians noted that the opioid risk alert was particularly helpful in adjusting the prescriptions they write. Ultimately, physicians were pleased with how EDSN alerts could improve connectedness across the health care continuum.

Finally, physicians believed that both improving the actual alert and improving physicians' understanding of it could prove beneficial. As one provider asserted, “Some sort of education along the lines of what you use that for would be helpful. I mean, you get the notification and the information but if there is no clinical utility or action it just becomes extra information that is easy to ignore” (AP 11). By including more tailored information and a clearer call-to-action and providing additional educational resources to aid physicians, respondents believed EDSN can achieve its potential.

Discussion

Ultimately, we found providers hold largely neutral or positive views regarding EDSN. Prior to implementation, physicians worried about alerts' potential to restrict their autonomy. While some concerns with effectiveness remained postlaunch, physicians tended to believe EDSN is correctly calibrated, nonintrusive, and has the potential to supplement their clinical workflow and decision-making. This response did not appear to be tied to a clinician's previous knowledge of CDS technology, implying that EDSN's design and intention are intuitive regardless of a provider's technological literacy.

While few noted the notifications would produce a significant change in their clinical decision-making, some described them as helpful reminders. Further, none of the interviewees regarded EDSN alerts as detrimental. This is important, as physicians mentioned that consistently using these alerts depends on CDS trust and usability. These sentiments were consistent between those who were interviewed twice and those who were interviewed once.

Physicians largely believed the alert was correctly calibrated and fired at the appropriate times. Even respondents who suggested that EDSN was too sensitive acknowledged that this is better than the alternative due to the need to closely monitor these high-risk patients. This logic extends to sentiments regarding alert placement and design, as physicians believed that EDSN's level of intrusiveness is appropriate given the alert's important content. Further, the suggestion of using Emojis to represent levels of severity aligns with other literature suggesting their potential value.[29] [30] [31]

While EDSN is not viewed negatively, alert use depends on a user's experience with all alerts within the EHR.[32] Thus, administrators should remain vigilant of the number of live EHR alerts at any given moment to minimize alert fatigue. Further, it is important to acknowledge that ambivalent feelings toward the alert may be tied to the alert's nonintrusiveness. If this alert was more intrusive, perhaps clinicians would harbor more negative sentiments. Interviewees' comments on improving comprehensiveness, however, implied that sacrificing the alerts subtlety for more comprehensiveness would be a desirable trade-off.[33]

This greater detail may include more information on where the patient recently received care, care frequency within a defined timeframe, previously filled prescriptions, and prior laboratory results. While alert additions may prove beneficial, care must be taken to avoid introducing algorithmic bias into EDSN, potentially harming those the algorithm aims to help.[16]

The success of health informatics projects depends on a comprehensive rollout accompanied with a helpful explanation of the alert and its capabilities. One-off efforts to make people aware of a new alert can be easily overlooked or forgotten.[21] To improve awareness, additional educational resources could prove helpful for both new and experienced physicians. Previous literature has shown this continual interplay to be crucial in improving CDS system uptake.[11] [22] [34] [35] While the EDSN launch was accompanied with an initial explanation, subsequent refreshers may be needed, particularly as EDSN data are gathered and hospitals experience provider turnover.

This project has several limitations. First, physician interviews occurred off-shift. Thus, physicians may have suffered from recall bias, misremembering true alert frequency. Next, given this was a convenience sample, there is potential for selection bias based on those who chose to respond. If bias here is significant, these findings may not represent the entire ED physician population. Also, a minority of interviewees were interviewed both before and after intervention, limiting longitudinal understandings. Further, this research only focused on one health system, limiting generalizability. Additionally, these interviews occurred over the course of a year, and time trends affecting alert familiarity may exist. Finally, COVID-19 onset may have skewed responses by deprioritizing attention paid to non-COVID complications. While this is difficult to fully evaluate, we paused interviews during COVID waves to capture sentiments in times most unaffected by the pandemic.

Despite these limitations, this research holds promise in helping informatics teams design better CDS systems. Future research will quantify EDSN's impact on ED visits, opioid overdose changes, prescribing practices, and their associated health care costs. The results of this and ongoing research will inform EDSN's continued implementation.

Conclusion

EDSN appears to have a marginally positive effect on physician workflow. At its most beneficial, the alert reminds physicians to consider the drugs they were prescribing to high-risk populations and to potentially adjust their care and referrals. At its least beneficial, the alert does not affect physician decision-making but is not intrusive to the point of negatively impacting physicians' workflow.

Further education and awareness could promote uptake and allow for ongoing physician input. Policymakers and health system decision-makers should incentivize early and consistent interplay between CDS designers and users. Gauging physician sentiments is crucial to understanding how these alerts can best help the physician, and by extension, the patient.

Clinical Relevance

This work evaluates the implementation of a CDS system before and after its launch within the EHR, through the lens of provider sentiments. Understanding physicians' role in the development and use of these systems is crucial as they continue to proliferate health care.

Multiple Choice Questions

1. For what adverse events does EDSN estimate a patient's likelihood of experiencing?

   • Sepsis and opioid risk

   • Future ED use and opioid risk

   • Sepsis and inpatient admission

   • Future ED use and head injury severity

   Correct answer: The correct answer is option b. This technology identifies patients at elevated risk for both high levels of future ED use and adverse outcomes due to opioid use.

2. In the preimplementation interviews which of the following was listed as a benefit of CDS technology?

   • Legal liability

   • Loss of physician agency

   • Lack of trustworthiness

   • Contained actionable recommendations

   Correct answer: The correct answer is option d. The first three choices were all listed as concerns, while the last choice was listed as a potential strength of the technology

3. What is the overarching sentiment regarding the EDSN alert (postimplementation)?

   • Overwhelmingly positive

   • Overwhelmingly negative

   • Mostly neutral but leaning positive

   • Mostly neutral but leaning negative

   Correct answer: The correct answer is option c. The consistent thread through the 12 postimplementation responses suggested that EDSN largely did not influence their clinical workflow but it did not negatively affect them. With iterations and a more comprehensive alert, providers believed EDSN could reach its potential.

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

Study participants were deidentified prior to analysis, with identifying information contained in a password-protected file within a secure file-sharing environment. This study was reviewed and approved by the relevant Institutional Review Board. Study participants were compensated for their participation via gift card.

• References

• 1 Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med 2020; 3: 17
• 2 Shah SN, Amato MG, Garlo KG, Seger DL, Bates DW. Renal medication-related clinical decision support (CDS) alerts and overrides in the inpatient setting following implementation of a commercial electronic health record: implications for designing more effective alerts. J Am Med Inform Assoc 2021; 28 (06) 1081-1087
• 3 Rosenthal B, Skrbin J, Fromkin J. et al. Integration of physical abuse clinical decision support at 2 general emergency departments. J Am Med Inform Assoc 2019; 26 (10) 1020-1029
• 4 Fletcher GS, Aaronson BA, White AA, Julka R. Effect of a real-time electronic dashboard on a rapid response system. J Med Syst 2017; 42 (01) 5
• 5 Whitcomb WF, Lucas JE, Tornheim R, Chiu JL, Hayward P. Association of decision support for hospital discharge disposition with outcomes. Am J Manag Care 2019; 25 (06) 288-294
• 6 Amland RC, Hahn-Cover KE. Clinical decision support for early recognition of sepsis. Am J Med Qual 2019; 34 (05) 494-501
• 7 Henry KE, Hager DN, Pronovost PJ, Saria S. A targeted real-time early warning score (TREWScore) for septic shock. Sci Transl Med 2015; 7 (299) 299ra122
• 8 Jenssen BP, Kelleher S, Karavite DJ. et al. A clinical decision support system for motivational messaging and tobacco cessation treatment for parents: pilot evaluation of use and acceptance. Appl Clin Inform 2023; 14 (03) 439-447
• 9 Shear K, Rice H, Garabedian PM. et al. Usability testing of an interoperable computerized clinical decision support tool for fall risk management in primary care. Appl Clin Inform 2023; 14 (02) 212-226
• 10 Levin S, Toerper M, Hamrock E. et al. Machine-learning-based electronic triage more accurately differentiates patients with respect to clinical outcomes compared with the emergency severity index. Ann Emerg Med 2018; 71 (05) 565-574.e2
• 11 Amarasingham R, Patzer RE, Huesch M, Nguyen NQ, Xie B. Implementing electronic health care predictive analytics: considerations and challenges. Health Aff (Millwood) 2014; 33 (07) 1148-1154
• 12 Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff (Millwood) 2014; 33 (07) 1123-1131
• 13 Blackburn J, Ousey K, Goodwin E. Information and communication in the emergency department. Int Emerg Nurs 2019; 42: 30-35
• 14 Geissert P, Hallvik S, Van Otterloo J. et al. High-risk prescribing and opioid overdose: prospects for prescription drug monitoring program-based proactive alerts. Pain 2018; 159 (01) 150-156
• 15 Weiner SG, Baker O, Bernson D, Schuur JD. One-year mortality of patients after emergency department treatment for nonfatal opioid overdose. Ann Emerg Med 2020; 75 (01) 13-17
• 16 Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 2019; 366 (6464) 447-453
• 17 Manrai AK, Funke BH, Rehm HL. et al. Genetic misdiagnoses and the potential for health disparities. N Engl J Med 2016; 375 (07) 655-665
• 18 Gijsberts CM, Groenewegen KA, Hoefer IE. et al. Race/ethnic differences in the associations of the Framingham risk factors with carotid IMT and cardiovascular events. PLoS One 2015; 10 (07) e0132321
• 19 Hussain MI, Reynolds TL, Zheng K. Medication safety alert fatigue may be reduced via interaction design and clinical role tailoring: a systematic review. J Am Med Inform Assoc 2019; 26 (10) 1141-1149
• 20 Safi S, Thiessen T, Schmailzl KJ. Acceptance and resistance of new digital technologies in medicine: qualitative study. JMIR Res Protoc 2018; 7 (12) e11072
• 21 Benda NC, Das LT, Abramson EL. et al. “How did you get to this number?” Stakeholder needs for implementing predictive analytics: a pre-implementation qualitative study. J Am Med Inform Assoc 2020; 27 (05) 709-716
• 22 Jeffery AD, Novak LL, Kennedy B, Dietrich MS, Mion LC. Participatory design of probability-based decision support tools for in-hospital nurses. J Am Med Inform Assoc 2017; 24 (06) 1102-1110
• 23 Fernandes M, Vieira SM, Leite F, Palos C, Finkelstein S, Sousa JMC. Clinical decision support systems for triage in the emergency department using intelligent systems: a review. Artif Intell Med 2020; 102: 101762
• 24 Weiss A, Jiang HJ. Overview of Clinical Conditions With Frequent and Costly Hospital Readmissions by Payer, 2018. Healthcare Cost & Utilization Project: Agency for Healthcare Research and Quality (AHRQ); 2021
• 25 Soares III WE, Melnick ER, Nath B. et al. Emergency department visits for nonfatal opioid overdose during the COVID-19 pandemic across six US Health Care Systems. Ann Emerg Med 2022; 79 (02) 158-167
• 26 Shortliffe EH, Sepúlveda MJ. Clinical decision support in the era of artificial intelligence. JAMA 2018; 320 (21) 2199-2200
• 27 Nowell LS, Norris JM, White DE, Moules NJ. Thematic analysis: striving to meet the trustworthiness criteria. Int J Qual Methods 2017; 16 (01) 1-13
• 28 Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006; 3 (02) 77-101
• 29 Pourmand A, Quan T, Amini SB, Sikka N. Can emoji's assess patients' mood and emotion in the emergency department? An emoji based study. Am J Emerg Med 2020; 38 (04) 842-843
• 30 Lai D, Lee J, He S. Emoji for the medical community-challenges and opportunities. JAMA 2021; 326 (09) 795-796
• 31 Skiba DJ. Face with tears of joy is word of the year: are emoji a sign of things to come in health care?. Nurs Educ Perspect 2016; 37 (01) 56-57
• 32 Bakken S. Need for innovation in electronic health record-based medication alerts. J Am Med Inform Assoc 2019; 26 (10) 901-902
• 33 Miller SD, Murphy Z, Gray JH. et al. Human-centered design of clinical decision support for anemia screening in children with inflammatory bowel disease. Appl Clin Inform 2023; 14 (02) 345-353
• 34 Levy-Fix G, Kuperman G, Elhadad N. Machine Learning and Visualization in Clinical Decision Support: Current State and Future Directions. arXiv preprint; 2019
• 35 Chan CT, Carlson J, Lee T, Vo M, Nasr A, Hart-Cooper G. Usability and utility of human immunodeficiency virus pre-exposure prophylaxis clinical decision support to increase knowledge and pre-exposure prophylaxis initiations among pediatric providers. Appl Clin Inform 2022; 13 (05) 1141-1150

Publikationsverlauf

Eingereicht: 27. Februar 2023

Angenommen: 06. Juni 2023

Artikel online veröffentlicht:
06. September 2023