Association between Electronic Health Record Implementations and Hospital-Acquired Conditions in Pediatric Hospitals

Keywords

quality - process improvement - electronic health records - safety - pediatrics - implementations and transitions

Background and Significance

Electronic health records (EHR) are nearly ubiquitous in health systems across the country,[1] [2] [3] [4] with many institutions having recently undergone or planning to undergo transitions between EHR systems.[5] [6] [7] Implementing an EHR is one of the most complex and operationally disruptive tasks a health system can undertake[7] and, as such, carries the potential to impact patient safety.[8] [9] For example, workflow disruptions or distractions such as provider training may negatively impact the quality of care in the phase immediately before or after such a transition. Along these lines, there are conflicting single-site reports of patient safety issues following EHR implementations and transitions.[10] [11] [12] [13] [14] While a recent study found no increase in patient safety events following EHR transitions in a cohort of adult hospitals,[15] there is limited evidence beyond dated, single-site studies in pediatric hospitals, which carry their own set of unique operational health technology challenges.

We sought to better understand the impact of EHR implementations on patient safety at pediatric institutions. To achieve this aim, we utilized data from Solutions for Patient Safety (SPS), a network of 145+ children's hospitals that share data and protocols to reduce harm in pediatric healthcare delivery.[16] [17] We studied the association between hospital-acquired condition (HAC) event rates and preventive care bundle compliance rates in the time immediately leading up to and after an EHR implementation at a cohort of SPS collaborators.

Methods

To study the impact of EHR implementations on patient safety in pediatric hospitals, we conducted a retrospective cohort study of 27 hospitals in the SPS network. Monthly HAC event and associated preventive care bundle compliance rates were analyzed in the months preceding and succeeding an EHR implementation. This study received a determination of “not human subjects research” by the Stanford University Institutional Review Board due to the aggregated and deidentified nature of the data.

Data Acquisition

We conducted a survey of EHR vendors and information technology (IT) leaders to identify EHR implementations occurring at children's hospitals between 2012 and 2022. This list was cross-referenced with the SPS database to create an anonymized dataset of monthly HAC rates and bundle compliance rates in the 7 months preceding and succeeding the transition. Data were available for 43 institutions. To preserve anonymity, analysis was done using dates relative to EHR go-live (months: −7 to +7). Both first-time EHR implementations and transitions from prior EHR systems were included. We are unable to determine exactly which sites underwent a transition from a prior EHR implementation compared with a first-time EHR implementation due to the anonymization process and missing information in survey responses. Given the observation period, however, we suspect that the majority of included sites experienced a transition from a prior EHR system to another one. For the purposes of the analysis, these two scenarios are considered together as an “EHR implementation.”

Six HACs were included for analysis: central-line associated bloodstream infection (CLABSI), catheter-associated urinary tract infection (CAUTI), adverse drug events (ADE), surgical site infections (SSI), pressure injuries (PI), and falls ([Table 1]). Operational definitions used by SPS for these measurements have been previously published by Lyren et al.[16] Eight associated bundle compliance rates ([Table 1]) were also considered; however, only four were included for final analysis due to limited data availability.

Table 1 Hospital-acquired condition and bundle compliance rate dataset completeness

Hospital-acquired condition	Normalized event rate measurement	Data completeness	Associated bundle compliance	Data completeness
CLABSI[a]	Events per 1,000 central-line days	100%	CLABSI insertion	50%
			CLABSI maintenance[a]	81%
SSI[a]	Events per 100 surgeries	99%	SSI[a]	61%
CAUTI[a]	Events per 1,000 catheter-days	97%	CAUTI insertion	37%
			CAUTI maintenance[a]	66%
Falls[a]	Events per 1,000 patient-days	96%	Falls	41%
PI[a]	Events per 1,000 patient-days	92%	PI[a]	60%
ADE[a]	Events per 1,000 patient-days	80%	ADE	28%

Abbreviations: ADE, adverse drug events; CAUTI, catheter-associated urinary tract infection; CLABSI, central-line associated bloodstream infection; PI, pressure injuries; SSI, surgical site infections.

^a Included in final dataset for analysis.

Inclusion Criteria

Only institutions with full data (all 15 months) for at least one HAC were included for analysis. This yielded a total of 27 out of the original 43 eligible institutions. For these remaining institutions, data completeness (the proportion of months for which a measurement was available) was calculated for six HACs and eight associated bundles ([Table 1]). Given the limited availability of data for many bundle compliance rates, only four bundles were included for final analysis, including CLABSI maintenance bundle, CAUTI maintenance bundle, SSI bundle, and PI bundle, as those were the measurements with greater than 50% data completeness.

Data Analysis

Monthly normalized HAC and bundle compliance rates were plotted against time relative to EHR implementation (month 0) and visually inspected. To determine if there was a statistically significant temporal association between these measurements and the time immediately surrounding EHR implementations, the observation period was divided into three eras of equal length: “before” (months: −7 to −3), “during” (months: −2 to +2), and “after” implementation (months: +3 to +7). The decision was made to include both the time immediately preceding and succeeding the EHR implementation in the “during” era. In our operational experience, much of the disruption associated with EHR implementation occurs in the time leading up to transition with provider training and other preparations. This also matches frameworks employed in prior studies around EHR implementations, which identify the three stages of preimplementation, implementation, and postimplementation.[18] Average monthly HAC rates were calculated across eras. Two-sided paired t-tests were performed to compare HAC rates between the eras. For each measurement of interest, sites were excluded from the paired t-test if they did not have data across all three eras. To correct for multiple hypothesis testing, the Bonferroni method was used to calculate p-value thresholds that would assure a 5% risk of type I error.[19] This resulted in p-value thresholds of 0.0028 for HACs and 0.0042 for bundle compliance. This process was repeated to perform subgroup analyses based on hospital size and year of implementation. Due to data restrictions to preserve the anonymity of participating sites, additional hospital characteristics were unavailable. Subgroups were stratified in a binary matter. For hospital size, a bed size of 200 was used as a binary cut-off between the two groups. For EHR implementation date, the year 2016 was used as a binary cut-off between the two groups.

Results

Visual inspection of the data revealed no increase in six average HAC rates (CLABSI, CAUTI, ADE, SSI, PI, and falls) associated with the time period immediately surrounding an EHR implementation ([Fig. 1]). Similarly, there was also no decrease observed for the four related bundle compliance rates: CLABSI maintenance bundle, CAUTI maintenance bundle, SSI bundle, and PI bundle ([Fig. 2]). Analysis of average HAC and bundle compliance rates by era (before, during and after EHR implementation) similarly revealed no statistically significant change between time periods ([Table 2]). None of the six analyzed HACs showed an increase in average rate the “during” EHR implementation period. While bundle compliance rates were observed to decrease slightly in the “during” EHR implementation period, these changes were minimal and not statistically significant. Subgroup analysis by year of implementation and hospital size also did not observe any statistically significant change during the EHR transition period. Of the 27 included sites, 14 had a size of <200 beds and 13 had a size ≥200 beds. Similarly, 10 sites experienced their EHR transition before 2016 and 17 had their EHR implementation during or after 2016.

Table 2 (top) Average HAC and bundle compliance rates across EHR implementation era and (bottom) p-values from paired t-test comparing average HAC and bundle compliance rates across EHR implementation era

	Hospital-acquired conditions						Bundle compliance
Era	CLABSI (N = 27)	CAUTI (N = 27)	ADE (N = 21)	SSI (N = 26)	PI (N = 26)	Falls (N = 26)	CLABSI (N = 22)	CAUTI (N = 18)	ADE (N = 6)	SSI (N = 16)
	Average normalized monthly event rates[a]						Bundle compliance rates
Before	1.30	1.32	0.05	3.00	0.09	0.04	0.87	0.85	0.90	0.88
During	1.16	0.81	0.05	1.72	0.09	0.03	0.86	0.84	0.86	0.85
After	1.16	2.46	0.11	2.25	0.10	0.01	0.86	0.83	0.85	0.84
	p -Values
Before vs. during	0.40	0.24	0.33	0.02	0.87	0.86	0.46	0.68	0.31	0.82
During vs. after	0.99	0.11	0.08	0.25	0.72	0.22	0.78	0.91	0.38	0.68
Before vs. after	0.30	0.26	0.13	0.20	0.89	0.02	0.82	0.94	0.43	0.49

Abbreviations: ADE, adverse drug events; CAUTI, catheter-associated urinary tract infection; CLABSI, central-line associated bloodstream infection; PI, pressure injuries; SSI, surgical site infections.

^a Normalized rates are events per 1,000 line-days for CLABSI and CAUTI; events per 1,000 patient-days for PI, ADE, and Falls; and events per 100 surgeries for SSI.

Fig. 1 Scatter plot of normalized monthly event rates for hospital-acquired conditions relative to electronic health record (EHR) implementation month for 27 included sites. Orange line represents the average monthly rate across all sites. Dotted gray lines separate the three eras: “before,” “during,” and “after” EHR implementation.

Fig. 2 Scatter plot of monthly compliance rates for preventive care bundles relative to electronic health record (EHR) transition month for 27 included sites. Orange line represents the average monthly rate across all sites. Dotted gray lines separate the three eras: “before,” “during,” and “after” EHR implementation.

Discussion

This observational multisite study discovered no meaningful increase in HAC rates and similarly no decrease in preventive care bundle compliance rates around EHR implementations at pediatric institutions. The ability to systematically analyze the experience of multiple children's hospitals undergoing an EHR implementation within the last decade, as this study does, is an important advancement in understanding the potential effect of the operationally disruptive but necessary task of transitioning to a new EHR. The findings from this study can inform and prepare children's hospitals as they plan or consider future EHR implementations.

Electronic Health Records Transitions and Adverse Events

Our findings are consistent with a 2016 study, which also found no association between EHR implementations and adverse patient outcomes at adult hospitals.[15] That study by Barnett et al used Medicare data from 2011 to 2012 to study both proximal (e.g., HAC rates, as captured by the Medicare patient safety indicator PSI-90) and more distal patient outcomes (e.g., readmission rates and mortality rates). In addition to focusing on adult hospitals, another important distinction is that Barett et al specifically studied the 90 days immediately following an EHR implementation, whereas our study analyzes the period of time surrounding the EHR implementation, including the months immediately preceding and succeeding it. In our operational experience overseeing EHR implementations at our own institution, this is the maximally disruptive period.

The findings from our study extend the limited literature on EHR implementations and patient harm, especially in pediatrics. To date, this literature consists mostly of conflicting single-site experiences and reviews of anecdotal reports. For example, one of the first such studies was by Han et al, who reported an increase in patient mortality in the period following their transition to computerized provider order entry in late 2002.[10] This was followed by studies from other children's hospitals which observed no change in patient mortality[11] and an overall decrease in mortality.[12]

An important distinction is that these early reports consist of experiences implementing EHR systems for the first time. However, the operational task facing most institutions in this era is transitioning from one EHR to another, which one would expect to pose an overlapping but slightly different set of challenges.[7] This makes recent reports, such as that by Whalen et al, more relevant to operational informatics leaders planning an EHR transition in the current era.[13] In their recent study, Whalen et al observed an increase in pediatric medication safety errors immediately following a transition to a new EHR system, which was resolved by 3 months after the transition. Their experience also highlights the unique operational challenges of pediatric institutions, especially those operating within a larger adult-focused health system, where the EHR may not be optimized for pediatric-specific tasks such as weight-based medication dosing.

Implications for Clinical Operations

The implementation of a new EHR is a complex and disruptive operational process with implications for patient safety. Potential sources of patient harm in the implementation process include disruption to workflows, disruption in patient data migration,[20] and changes in clinical decision support systems,[13] [21] among other factors. As such, the negative findings of our study may be surprising. This observation likely represents a broader maturity of the health care system in executing the EHR implementation process and mitigating safety impacts, as we learn from our collective experiences.

As EHRs have become nearly ubiquitous, there is a growing body of research around understanding and mitigating patient harm during EHR implementations. For example, there are several recent studies aimed at developing frameworks to categorize and characterize patient safety events that occur around EHR transitions. This includes a study by Meeks et al in which interviews with personnel involved with EHR implementations at 12 different hospitals were qualitatively analyzed to construct an eight-part framework for categorizing safety events around EHR implementations.[8] Similarly, Patterson et al also report qualitative analysis findings from a survey of health care IT professionals eliciting experiences with patient safety events around EHR transitions.[9] They clustered events into 13 domains, assigning priorities to the various categories of safety risks. Although not specifically tied to EHR transitions, Menon et al report the use of “safety huddles” to identify and address EHR-related safety risks early.[22] Such a method can be used to quickly identify safety issues as they arise through the implementation process.

These frameworks combined with reviews summarizing best practices and lessons learned from recent EHR transitions, such as those by Boonstra et al, Saleem and Herout, and Huang et al, serve as valuable resources to those planning to undertake an EHR implementation.[7] [14] [23]

Limitations

There are several important limitations of this study. While the ability to pool data from multiple institutions is a strength of the study, the sample size of 27 sites limits the ability to detect smaller changes in event rates, especially within subgroup analyses. This is especially true for HAC event and bundle compliance rates with missing data, such as ADEs, for which there was only data to compare six sites in the paired t-tests. The variance in HAC event rates similarly limits the study's power.

Additionally, the anonymized and limited nature of the SPS dataset restricts the complexity of the analysis that can be done. To preserve the identity of SPS sites regarding sensitive patient safety information, we are unable to do more nuanced subgroup or multivariable analyses for factors of potential interest. For example, it is possible that EHR implementations that occurred in a standalone hospital may differ from those that are part of a larger adult-focused health system and that when blended together, a true association becomes undetectable. While we did perform subgroup analysis with binary stratification by EHR implementation year and hospital size, the small subgroup size limits the power of these analyses. We are also unable to distinguish first-time EHR implementations from transitions from legacy system. Similarly, because of the nature of the data source, we are also unable to assign a cohort of “control” hospitals, which could have been used to adjust for secular trends and other confounding factors.

Finally, this study is focused on the inpatient care setting and on one specific type of patient harm, HACs, as those are the types of adverse events captured by the SPS collaborative. Measurements of safety in the ambulatory care setting and other inpatient outcomes of interest (e.g., mortality, readmission, serious safety events, etc.) were outside the scope of this study. It is important to note that HACs comprise only a portion of patient harm, and future work may benefit from more comprehensive measures of patient safety.[24] Along those lines there are additional approaches to estimating patient harm through other methods, including the use of EHR process measurements, the Global Assessment of Pediatric Patient Safety Trigger Tool, and other health IT frameworks.[25] [26] [27] [28] Ensuing research may explore these additional measures to better understand the broader question around immediate disruption from EHR implementations and their effect on patient safety. We envision a future study that leverages a more comprehensive dataset of adverse events and other clinical outcomes, collected through a multisite collaboration focused specifically on this topic. In the meantime, given the obstacles to obtaining such data from multiple institutions, this study is an important addition to the literature despite these limitations.

Conclusion

This multisite study of 27 children's hospitals detected no overall increase in HACs event rates and no overall decrease in preventive care bundle compliance rates in the months surrounding an EHR implementation. These findings extend the growing body of literature characterizing and quantifying the immediate effect of EHR implementations on patient safety outcomes.

Clinical Relevance

This manuscript studies the immediate impact of EHR implementation on patient safety, which is of core relevance to the use of information technology to support patient care.

Multiple Choice Questions

1. Which of the following patient safety/quality concepts is best described as a collection of evidence-based practices that, when performed reliably, can improve processes of care and patient outcomes?

   • Verbal nursing orders

   • Care bundle

   • Rounding checklist

   • Medication reconciliation

   Correct Answer: The correct answer is option b. Care bundles are a series of evidence-based practices, which when performed collectively and reliably, improve processes of care and promote patient safety. Typically care bundles are applied to specific processes, such as central line insertion, surgical site care, etc. Rounding checklists and medication reconciliation are related practices that can improve patient safety when adopted, but are distinct from care bundles.

2. This study found no significant change in which patient type of safety measure in the time immediately surrounding an EHR implementation?

   • Readmission

   • Mortality

   • Hospital-acquired conditions

   • Care gaps

   Correct Answer: The correct answer is option c. This study detected no overall increase in hospital-acquired conditions event rates and no overall decrease in preventive care bundle compliance rates in the months surrounding an EHR implementation. Hospital-acquired conditions are an important measure of patient safety. Hospital-acquired conditions refer to medical conditions or complications that occur during a hospitalization. Some examples of hospital-acquired conditions include central-line associated bloodstream infections, surgical site infections, and adverse drug events. A related study by Barnett et al did incorporate measures of patient mortality and readmission, but those measures were outside of the scope of this study.

• References

• 1 Nakamura MM, Harper MB, Jha AK. Change in adoption of electronic health records by US children's hospitals. Pediatrics 2013; 131 (05) e1563-e1575
• 2 Charles D, Gabriel M, Furukawa M. Adoption of Electronic Health Record Systems among U.S. Non-federal Acute Care Hospitals: 2008–2013. ONC Data Brief [Internet]. 2014 May; Accessed April 27, 2023: https://www.healthit.gov/sites/default/files/oncdatabrief16.pdf
• 3 Kruse CS, Kristof C, Jones B, Mitchell E, Martinez A. Barriers to electronic health record adoption: a systematic literature review. J Med Syst 2016; 40 (12) 252
• 4 Adler-Milstein J, Holmgren AJ, Kralovec P, Worzala C, Searcy T, Patel V. Electronic health record adoption in US hospitals: the emergence of a digital “advanced use” divide. J Am Med Inform Assoc 2017; 24 (06) 1142-1148
• 5 Lammers EJ, Zheng K. Characteristics associated with hospital health IT vendor switching and dropping. AMIA Annu Symp Proc 2011; 2011: 742-749
• 6 Adler KG, Edsall RL. EHR switch survey: responses from 305 family physicians. Fam Pract Manag 2015; 22 (01) 13-18
• 7 Huang C, Koppel R, McGreevey III JD, Craven CK, Schreiber R. Transitions from one electronic health record to another: challenges, pitfalls, and recommendations. Appl Clin Inform 2020; 11 (05) 742-754
• 8 Meeks DW, Takian A, Sittig DF, Singh H, Barber N. Exploring the sociotechnical intersection of patient safety and electronic health record implementation. J Am Med Inform Assoc 2014; 21 (e1): e28-e34
• 9 Patterson ES, Anders S, Moffatt-Bruce S. Clustering and prioritizing patient safety issues during EHR implementation and upgrades in hospital settings. Proc Int Symp Hum Factors Ergon Healthc 2017; 6 (01) 125-131
• 10 Han YY, Carcillo JA, Venkataraman ST. et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116 (06) 1506-1512
• 11 Del Beccaro MA, Jeffries HE, Eisenberg MA, Harry ED. Computerized provider order entry implementation: no association with increased mortality rates in an intensive care unit. Pediatrics 2006; 118 (01) 290-295
• 12 Longhurst CA, Parast L, Sandborg CI. et al. Decrease in hospital-wide mortality rate after implementation of a commercially sold computerized physician order entry system. Pediatrics 2010; 126 (01) 14-21
• 13 Whalen K, Lynch E, Moawad I, John T, Lozowski D, Cummings BM. Transition to a new electronic health record and pediatric medication safety: lessons learned in pediatrics within a large academic health system. J Am Med Inform Assoc 2018; 25 (07) 848-854
• 14 Saleem JJ, Herout J. Transitioning from one electronic health record (ehr) to another: a narrative literature review. Proc Hum Fact Ergon Soc Annu Meet. 2018; 62 (01) 489-493
• 15 Barnett ML, Mehrotra A, Jena AB. Adverse inpatient outcomes during the transition to a new electronic health record system: observational study. BMJ 2016; 354: i3835
• 16 Lyren A, Brilli RJ, Zieker K, Marino M, Muething S, Sharek PJ. Children's hospitals' solutions for patient safety collaborative impact on hospital-acquired harm. Pediatrics 2017; 140 (03) e20163494
• 17 Lyren A, Coffey M, Shepherd M, Lashutka N, Muething S. SPS Leadership Group. We will not compete on safety: how children's hospitals have come together to hasten harm reduction. Jt Comm J Qual Patient Saf 2018; 44 (07) 377-388
• 18 Keshavjee K, Bosomworth J, Copen J. et al. Best practices in EMR implementation: a systematic review. AMIA Annu Symp Proc 2006; 2006: 982
• 19 Shaffer JP. Multiple hypothesis testing. Annu Rev Psychol 1995; 46 (01) 561-584
• 20 Gettinger A, Csatari A. Transitioning from a legacy EHR to a commercial, vendor-supplied, EHR: one academic health system's experience. Appl Clin Inform 2012; 3 (04) 367-376
• 21 Wright A, Aaron S, Seger DL, Samal L, Schiff GD, Bates DW. Reduced effectiveness of interruptive drug-drug interaction alerts after conversion to a commercial electronic health record. J Gen Intern Med 2018; 33 (11) 1868-1876
• 22 Menon S, Singh H, Giardina TD. et al. Safety huddles to proactively identify and address electronic health record safety. J Am Med Inform Assoc 2017; 24 (02) 261-267
• 23 Boonstra A, Versluis A, Vos JFJ. Implementing electronic health records in hospitals: a systematic literature review. BMC Health Serv Res 2014; 14: 370
• 24 Stockwell DC, Landrigan CP, Schuster MA. et al. Using a pediatric trigger tool to estimate total harm burden hospital-acquired conditions represent. Pediatr Qual Saf 2018; 3 (03) e081
• 25 Scharnweber C, Mollenkopf NL, Fackler J, Dover GJ, Lehmann CU. Utilizing electronic health record data to determine the health of the medication process after the relocation of a children's hospital. Stud Health Technol Inform 2013; 192: 210-214
• 26 Classen DC, Resar R, Griffin F. et al. ‘Global trigger tool’ shows that adverse events in hospitals may be ten times greater than previously measured. Health Aff (Millwood) 2011; 30 (04) 581-589
• 27 Stockwell DC, Landrigan CP, Toomey SL. et al; GAPPS Study Group. Adverse events in hospitalized pediatric patients. Pediatrics 2018; 142 (02) e20173360
• 28 Singh H, Sittig DF. Measuring and improving patient safety through health information technology: The Health IT Safety Framework. BMJ Qual Saf 2016; 25 (04) 226-232