Keywords
dashboard - guideline - performance
Background and Significance
Background and Significance
Quality improvement (QI) targets unwarranted variations in care.[1] National guidelines for bronchiolitis, the fourth most common reason for pediatric hospitalization in the United States, encourage limiting disproven treatments (e.g., antibiotics).[2]
[3] QI initiatives for bronchiolitis have proven effective[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12] but rely on timely data for audit and feedback.[13]
[14] Traditional methods to obtain data may be challenging in some organizations.[15]
[16]
At our organization we had several challenges contributing to variation in care for bronchiolitis, including (1) lack of accessible clinical guidelines, (2) minimal data support for QI initiatives, and (3) lack of organizational process for providing feedback to clinicians. In 2015 we joined a national QI initiative, “Stewardship in Improving Bronchiolitis,” from the American Academy of Pediatrics (AAP)[12] which provided tools and coaching to a local bronchiolitis QI workgroup and required close tracking of data to design targeted interventions. Our initial data request through the organization's data warehouse took 6 months to be completed, prompting us to review alternate options for acquiring data.
Visual analytics dashboards are one mechanism to overcome repeated manual electronic health record (EHR) queries to support QI integrating data into a user interface enabling tracking, planning, and comparisons with near real-time data from the EHR and other sources.[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25]
[26]
[27]
[28]
[29]
[30]
[31] Users may modify inclusions/exclusions to focus on their population of interest without repeat data requests.[28]
[29]
[30]
Objective
We aimed to describe the use of a visual analytics dashboard to support a multifaceted QI initiative for patients with bronchiolitis. We also intended to determine if the dashboard directly impacted the success toward aims of the multifaceted QI initiative, which were to achieve 20% reductions in the use of chest X-rays, bronchodilators, antibiotics, steroids, and viral testing in patients with bronchiolitis by April 2018.
Methods
The QI initiative took place at Children's Minnesota (CM), a large, independent, not-for-profit, tertiary children's health care organization with approximately 14,000 inpatient and 96,000 emergency department (ED) visits annually. There were five key interventions to the QI initiative. First, we joined a national bronchiolitis QI collaborative,[12] and second gathered a CM bronchiolitis workgroup (October 2015)—including representatives from ED, hospitalists, nursing, respiratory therapy, information technology (IT), pharmacy, and critical care—to determine local QI interventions. Third, clinicians received education (e.g., evidence behind limiting bronchodilators) at staff meetings and CM Grand Rounds January and October 2016. Nurses and respiratory therapists received education via modules and e-mail newsletters. Fourth, (February 2016) we published a local modification of the AAP 2014 bronchiolitis guideline[2] and a companion order-set to our intranet and EHR (Cerner[32]).
Following local QI interventions there were anecdotal improvements but data delays emerged as a key barrier to success; data requested in March 2016 was delivered in August 2016. As data turnaround was crucial for targeted interventions we sought an alternate method to obtain data. Workgroup members had used operational dashboards[33] (implemented 2011) and felt a clinical dashboard might improve bronchiolitis data procurement.
As a fifth QI intervention a pediatric hospitalist (bronchiolitis workgroup lead) and an IT dashboard developer partnered in modifying a vendor's analytic dashboard[33] for use in bronchiolitis. They determined target patient population based upon previous guidelines/studies,[2]
[34] categorized clinicians (ED vs. observation/inpatient [hereafter referred to as inpatient]), categorized tests/treatments (e.g., medications considered “antibiotics”), determined display metrics, benchmarks,[34] and verified data accuracy over ∼6 months. The hospitalist monitors the dashboard monthly during bronchiolitis season and works with the IT dashboard developer to resolve data accuracy or display issues.
Organizational leaders were granted dashboard access; feedback from early users was collected informally over 1 month and resulted in minor changes. Issues with accuracy of individual hospitalist data related to resident order entry led to limitation of individual log-ins to ED clinicians and organizational leaders.
The primary objective of this case study was to describe use of a bronchiolitis dashboard in the context of a multifaceted QI initiative. The process measure was the percentage of individual ED clinicians who logged in to the dashboard, obtained from IT records. QI outcome measures were percent use of chest radiographs, bronchodilators, antibiotics, steroids, and viral testing.[2] Balancing measures included length of stay (LOS), charge (hospital facility charges and professional fees presented as ratios per CM policy), and 7-day same-cause ED revisits or hospital readmissions.
Patients 2 months to 2 years old seen in ED/inpatient settings at CM with bronchiolitis (International Classification of Disease 9 or 10 codes 466.11, 466.19 or J21.x) in the period October 1, 2014 to April 30, 2018 were included. Data from May to September of each year were excluded as we suspected there may be appropriately higher use of nonrecommended tests/treatments outside of the typical bronchiolitis season and elected to focus QI interventions on peak season. Patients in the intensive care units or with a secondary diagnosis of asthma, pneumonia, or underlying complex chronic condition (including gestational age < 27 weeks)[35] were excluded.
We used statistical process control (SPC) p-charts for outcome measures using QI Macros (KnowWare International Inc., version 2018). October 2014 to April 2015 represented the baseline period for calculation of upper and lower control limits; a process change was indicated for the implementation period of October 2015 to April 2018 to calculate new control limits. We determined a priori[3] 20% reductions in the baseline metric mean to be clinically relevant, based upon our preimplementation QI aim. Measures were separated into ED/inpatient based upon ordering clinician. Resident/fellow orders are assigned to the attending clinician on record at the time of the order; this was not modifiable on the dashboard. Balancing measures were analyzed with chi-squared tests for dichotomous outcomes and independent t-tests for continuous outcomes using STATA version 13.0.[36] p-Values < 0.05 were statistically significant.
Clinicians were not required to review personal data compared with peers and there were no implications (e.g., financial) to their performance. There were no conflicts of interest. This study was deemed QI and exempt from further review by the CM's Institutional Review Board.
Results
We implemented a bronchiolitis dashboard in August 2016, see [Fig. 1]. Default settings restrict the dataset to the goal patient population; users can add a chart or change inclusions/exclusions to analyze specific populations/units. Green and red colors indicate performance better than or worse than the benchmark,[34] respectively. Target metrics can also be displayed by demographic categories, such as primary payer or race. Thirty-five percent (20/57) of ED clinicians logged in to the dashboard at least once.
Fig. 1 Sample views of the bronchiolitis visual analytic dashboard.
See [Fig. 2] for sample SPC charts for bronchodilator use with annotation of QI interventions; see online supplemental materials for SPC charts and summary table of percent change for all metrics ([Supplementary Figs. S1] and [S2], [Table S1], available in the online version). There was[3] 20% difference in all target metrics with the exception of ED and inpatient antibiotics and inpatient viral testing. For example, there was a shift in mean bronchodilator use from 66.7 to 43.8% in the ED and 72.1 to 46.4% in inpatients (33 and 36% reductions, respectively).
Fig. 2 (A, B) Statistical process control p-charts for bronchodilator use in patients with bronchiolitis. Baseline period: October 1, 2014 to April 30, 2015; implementation period: October 1, 2015 to April 30, 2018. CL, control limit (mean); LCL, lower control limit; UCL, upper control limit.
See [Table 1] for balancing measures. Comparing baseline with implementation periods, there were improvements in all ED balancing measures with a higher ED discharge rate (70.7 vs. 72.8%, p = 0.05), lower charges (ratio 1:0.86, p < 0.001), shorter LOS (2.9 vs. 2.6 hours, p = 0.001), and lower 7-day revisit rates (15.4 vs. 11.6%, p < 0.001). Inpatient charges increased (ratio 1:1.14, p = 0.01) but LOS and readmissions remained stable.
Table 1
Balancing measures across emergency department (ED) and observation/inpatient (admitted) care settings before (baseline) and after implementation of a multifaceted quality improvement initiative for patients with bronchiolitis
|
Baseline
(October 2014–April 2015)
|
Implementation
(October 2015–April 2018)
|
p-Value
|
|
Discharged home from ED, n (%)
|
N = 2,035 (70.7)
|
N = 3,965 (72.8)
|
0.05
|
|
Mean ED length of stay, hours (95% CI)
|
2.9 (2.7–3.1)
|
2.6 (2.5–2.6)
|
<0.001
|
|
Mean ED charges, ratio
|
1
|
0.86
|
<0.001
|
|
7-Day ED revisit, n (%)
|
313 (15.4)
|
458 (11.6)
|
<0.001
|
|
Admitted, n (%)
|
N = 843 (29.3)
|
N = 1,485 (27.2)
|
0.05
|
|
Mean inpatient length of stay, days (95% CI)
|
2.3 (2.2–2.4)
|
2.6 (2.4–2.8)
|
0.07
|
|
Mean inpatient charges, ratio
|
1
|
1.14
|
0.01
|
|
7-Day readmission, n (%)
|
8 (0.95)
|
10 (0.67)
|
0.37
|
Abbreviation: CI, confidence interval.
Two targeted interventions resulted from the use of the dashboard. First, in response to high inpatient bronchodilator use noted in April 2016 (68%) we developed an EHR bronchodilator order alert (implemented mid-month, October 2016). Second, we exported individual scorecards from the dashboard to all ED clinicians in November 2017 with personal 2016 to 2017 data compared with peers. This process was repeated in late March 2018 with data from the 2017 to 2018 season. We were unable to determine which ED clinicians reviewed their scorecard.
Discussion
Health care dashboards have been used to improve workflow, reduce preventable harms, and track metrics across multiple sites.[17]
[18]
[20]
[21]
[22]
[23]
[24]
[26]
[27]
[31]
[37] We used a bronchiolitis clinical dashboard to support a multifaceted QI initiative for bronchiolitis. While improvement shifts were seen in most target metrics it is unlikely, based upon timing and low individual use, that these improvements were a direct result of the dashboard. The dashboard was viewed by leaders as instrumental in tracking adherence to guideline metrics and was used to inform additional targeted interventions.
The low individual log-in rate (35%) found in our study may reflect low team engagement, a key for QI success,[38] and may have tempered improvements in outcome measures. We suspect that because the dashboard required a separate login from the EHR, individual clinicians may have perceived low ease of use[39]
[40] or were uninterested in their data.
Our data, as well as other QI studies without use of a dashboard, suggest that the other bronchiolitis QI interventions, such as the guideline and order-set, caused the observed improvements.[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12] However, QI is reliant on having on timely data to perform Plan-Do-Study-Act cycles.[41] We experienced difficulty obtaining data (6 months) driving our decision to develop a dashboard where data are loaded to the dashboard within approximately 2 to 4 days of patient discharge, enabling near real-time feedback and targeted interventions. This dashboard allowed us to understand the impact of the other QI interventions.
While we saw significant shifts in all metrics aside from antibiotic use and inpatient viral testing, there was a relatively greater improvement in both outcome and balancing measures in the ED versus inpatients. It is possible that interventions targeted to ED clinicians alone (individual dashboard log-in, individual scorecards) would have contributed to this difference; however, we suspect this did not occur as the timing of ED improvements relative to the inpatient setting suggests that other factors may have been more important, such as shifts in patient severity evidenced by a higher rate of ED discharge in implementation. There may also have been differences in guideline uptake, lack of individual hospitalist feedback, or difficulty in changing culture.[6]
[42]
[43]
[44]
[45] Future studies should examine reasons for bronchiolitis guideline nonadherence.
There were no clinically relevant changes in antibiotic use in either setting (14% decrease in ED, 14% increase in inpatients). By using the dashboard to add the option to exclude patients with otitis media the antibiotic rate was under 5%, indicating alternate appropriate indications for high antibiotic use not captured by original data definitions.
Balancing measure improvements included shorter ED LOS and lower ED charges in the implementation periods. This is consistent with a previous study which found that higher adherence to a bronchiolitis guideline was associated with lower LOS and costs.[46] Inpatient LOS was unchanged and inpatient charges were slightly higher in our study. Our QI initiative did not target hospital LOS—which largely correlates with charges/costs in the inpatient setting—as such these findings were not surprising.
Future directions for this QI initiative include increasing clinician engagement and feedback and use of the dashboard to design and track new group goals (e.g., reducing inpatient viral testing). Future initiatives may apply heuristic evaluation to improve dashboard visualization[47] and explore barriers to use.
Challenges included attributing orders to appropriate clinicians in a teaching environment and a moderate learning curve. We found it critical to partner a clinician with the IT dashboard developer to determine default settings and understand workflow limitations. Due to low individual use it is difficult to determine the impact of individual performance review or the direct impact of the dashboard on outcomes. A final limitation to this study is that the cost of the QI initiative, including the dashboard which is subject to vendor negotiation, was not directly measured.
Conclusion
We described use of a visual analytics dashboard in a multifaceted bronchiolitis QI initiative. Subsequent to multiple QI interventions we reduced use of most nonrecommended tests and treatments in patients with bronchiolitis and improved ED balancing measures. However, timing of improvements and low individual clinician use suggest that the dashboard did not directly impact outcomes. The dashboard was helpful in overcoming organizational barriers to QI data procurement and in tracking the impact of other QI interventions and may be considered as a tool for other organizations with similar challenges.
Clinical Relevance Statement
Clinical Relevance Statement
This case report has relevance for clinicians, IT specialists, and QI specialists as it describes the use of a visual analytics dashboard to inform QI initiatives and improve clinical care.
Multiple Choice Questions
Multiple Choice Questions
-
Which of the following may limit the ability to accurately attribute individual orders to individual attending clinicians on a visual analytics dashboard?
-
Clinician license type (e.g., MD vs. NP).
-
Attribution of resident orders.
-
Time of clinician order.
-
Type of order (e.g., medication vs. laboratory test).
Correct Answer: The correct answer is option b, attribution of resident orders. One challenge in our QI initiative was how to deal with resident orders. The organization EMR assigns resident orders to the attending clinician of record at the time the order was placed. We found this to be accurate in the ED setting but not in the inpatient (e.g., hospitalist) setting where the attending provider may change multiple times in a day.
-
Who might be the best partner for an IT dashboard developer when creating a visual analytics dashboard to support a clinical QI initiative?
-
A resident on a 1-month QI elective.
-
Another software programmer within the IT department.
-
A front-line clinical leader such as an ED clinician or hospitalist with QI expertise.
-
The chief financial officer.
Correct Answer: The correct answer is c, a front-line clinical leader such as an ED clinician or hospitalist with QI expertise. When bringing IT to the forefront of clinical work it is important to partner IT experts with front-line users and content experts. A lesson learned in our QI initiative was that it was crucial to have the IT developer partner with the QI clinical leader (a hospitalist) to review the dashboard, modify visual displays, and review data for validation.
