Variations among Electronic Health Record and Physiologic Streaming Vital Signs for Use in Predictive Algorithms in Pediatric Severe Sepsis

Abstract

Objective This study sought to describe the similarities and differences among physiologic streaming vital signs (PSVSs) and electronic health record (EHR)-documented vital signs (EVSs) in pediatric sepsis.

Methods In this retrospective cohort study, we identified sepsis patients admitted to the pediatric intensive care unit. We compared PSVS and EVS measures of heart rate (HR), respiratory rate, oxyhemoglobin saturation, and blood pressure (BP) across domains of completeness, concordance, plausibility, and currency.

Results We report 1,095 epochs comprising vital sign data from 541 unique patients. While counts of PSVS measurements per epoch were substantially higher, increased missingness was observed compared with EVS. Concordance was highest among HR and lowest among BP measurements, with bias present in all measures. Percent of time above or below defined plausibility cutoffs significantly differed by measure. All EVS measures demonstrated a mean delay from time recorded at the patient to EHR entry.

Conclusion We measured differences between vital sign sources across all data domains. Bias direction differed by measure, possibly related to bedside monitor measurement artifact. Plausibility differences may reflect the more granular nature of PSVS which can be critical in illness detection. Delays in EVS measure currency may impact real-time decision support systems. Technical limitations increased missingness in PSVS measures and reflect the importance of systems monitoring for data continuity. Both PSVS and EVS have advantages and disadvantages that must be weighed when making use of vital signs in decision support systems or as covariates in retrospective analyses.

Keywords

sepsis - vital signs - electronic health records - clinical decision support systems - physiologic

Introduction

Sepsis is a leading cause of mortality among children admitted to the pediatric intensive care unit (PICU).[1] Early detection of sepsis is critical, as the diagnosis is often delayed in the PICU[2] leading to delayed therapy and worse outcomes.[3] Risk prediction scores and machine learning models may predict the onset of sepsis in adults, children, and neonates.[4] [5] [6] From these algorithms, “early warning” clinical decision support (CDS) systems have sought to alert clinicians to patients at risk for developing sepsis.[7] [8] [9] Additionally, retrospective identification of sepsis patients through computable phenotypes provides important datasets for researchers.[10] [11] Notably a key feature in nearly all of these models is patient vital signs, including heart rate (HR), respiratory rate (RR), oxyhemoglobin saturation (SpO₂), and blood pressure (BP). The prediction accuracy of such models depends on the accuracy of recorded vital signs.

The widespread implementation of electronic health records (EHRs) has enabled electronic vital sign documentation through manual entry or verification from integrated patient monitors.[12] However, the datasets derived from electronic vital sign documentation are often incomplete or inconsistent.[13] [14] [15] Physiologic streaming vital signs (PSVSs) from patient bedside monitors offer an alternative to EHR-recorded vital signs (EVSs), providing more granular data in near-real time.[16] [17] However, significant technical challenges exist to developing streaming analytics platforms for use as CDS tools.[18] [19] [20]

Objective

Accurate vital sign measurement is essential to the detection and prediction of clinical deterioration in pediatric sepsis as well as in retrospective identification of sepsis patients. The goal of this study is to describe the similarities and differences among streaming vital signs (PSVS) versus documented vital signs (EVS) in PICU patients with sepsis. We hypothesize that PSVS and EVS will differ in the measured domains, based on known technical limitations of PSVS data storage as well as the entry and validation procedures required for EVS.

Methods

We designed a retrospective cohort study including patients admitted to a 55-bed single-center academic PICU from July 2017 through December 2020. This study was granted an exemption (# 19–016133, initial approval 5/24/2019, addendum approval 12/16/2020) by the Institutional Review Board at the Children's Hospital of Philadelphia.

We identified eligible encounters as those with a PICU admission and subsequent PICU discharge during the study period and an ICD-9 (International Classification of Diseases, 9th Revision) diagnosis of severe sepsis (995.92) or septic shock (785.52), as recorded in this site's Virtual Pediatric Systems (VPS) quality improvement database by trained data-entry nurses. These diagnosis codes have previously been shown to accurately identify sepsis encounters.[21] We extracted admission–discharge–transfer (ADT) events from the EHR (Epic Systems, Verona, Wisconsin, United States) database. We defined an ADT epoch as a bed assignment in the PICU with bounding ADT entry and exit events, within a single patient encounter. Epochs were excluded when the epoch duration was less than 1 hour or when the location was not a PICU bed (e.g., perioperative).

We collected demographics, associated diagnoses, length of stay, and patient outcome from our institutional VPS database. We extracted EVS for the variables including HR, RR, SpO₂, and BP, measured by noninvasive methods (noninvasive [cuff] BP [NBP]) or invasive arterial monitoring (arterial line [transduced] BP [ABP]). EVSs were either manually entered or integrated into the EHR from the bedside monitor network and then individually validated by the bedside nurse. PSVSs were broadcast from General Electric (GE) Solar 8000 bedside monitors on the GE CareScape network and captured by a Nuvon (Capsule Technologies, Andover, Massachusetts, United States) IDM 4000 interface, transmitted to an IBM InfoSphere stream server and stored in an IBM Hive database approximately every 5 seconds. In this study, we do not make use of continuous (waveform) physiologic signals, which increase data granularity at the expense of acquisition and storage complexity. Nonphysiologic (“implausible”) values were excluded from the dataset. Timestamps were assumed to be synchronous compared with EHR timestamps. We mapped EHR beds to corresponding bedside monitors and extracted the same vital sign variables as above for all available timestamps in each ADT epoch.

We analyzed ADT epochs with paired EVS and PSVS data across the domains of completeness, concordance, plausibility, and currency.[22] We report counts of vital signs by source, as well as median per epoch. We assessed completeness by measuring the number of hours of the ADT epoch and calculating the percent of hours without a vital sign recorded. Because NBP values are measured on the patient intermittently but bedside monitors continue to broadcast values continuously, we filtered the PSVS dataset to remove all NBP values which occurred >1 hour after they were measured on the patient (e.g., if the last measured NBP occurred at 10:15 a.m., we removed all NBP values which streamed from 11:15 a.m. until a new NBP was measured on the patient). This duration was chosen as it was the most common measurement interval for NBP.

To measure concordance between the two vital sign sources, we calculated the numeric distribution of the PSVS values within ± 20 minutes of each EVS recorded time and calculated the distribution mean and variance. We projected the EVS value onto this distribution and calculated its z-score position on the distribution. A positive z-score indicated that the EVS was larger (higher) than the mean of the PSVS distribution (“over-estimating”), while a negative z-score indicated that the EVS was smaller (lower) than the mean of the PSVS distribution (“under-estimating”).

We defined the plausibility of a vital sign value as whether the vital sign makes sense in light of what it is measuring. We applied age-specific cutoffs taken from a published consensus-derived sepsis pathway[23] which were derived from Goldstein et al.[24] The SpO₂ cutoff was fixed at 90% for all ages. We measured the total duration, or time until next vital sign measurement, and percent of time that vital signs fell above (or below) the consensus cutoff values. Based on our clinical population of interest, we considered time below SpO₂ and ABP cutoffs, and time above HR and RR cutoffs. We calculated the currency of EVS as the difference between the entry time and the recorded time. A positive difference indicated that the vital sign was entered in the EHR (“entry time”) after the time it was recorded from the patient (“recorded time”), whereas a negative currency indicates it was entered into the EHR prior to the time it was supposedly recorded from the patient. PSVSs had no measurable delay from bedside monitor to database storage and were considered current.

Descriptive statistics are reported as mean (standard deviation, SD) or median (interquartile range, IQR). Distributions are compared using t-tests and ANOVA (analysis of variance) when normality is assumed, or Wilcoxon signed rank and Kruskal–Wallis test when not normally distributed. Unless otherwise stated, a value of p <0.05 is considered significant. All analyses occurred in R studio[25] and made use of the packages odbc,[26] dbplyr,[27] dplyr,[28] ggplot2,[29] cowplot,[30] and knitr.[31]

Results

A total of 611 PICU encounters met inclusion criteria (4.5% of all PICU encounters during the study period), which included 541 unique patients and 1,095 ADT epochs. After removing epochs less than 60 minutes' duration and department locations not in the PICU, our resulting analysis included 808 ADT epochs from 608 PICU encounters representing 539 unique patients ([Fig. 1]). The median duration of these ADT epochs was 112 hours (IQR: 31 to 305). Patient and encounter demographics and outcomes are shown in [Table 1].

Table 1 Patient and encounter demographic information, outcomes, and common associated diagnoses

Patient demographics	Subjects (N = 539)
Gender: female (%)	263 (48.8)
Race[a] (%)
Asian	19 (3.5)
Black or African American	136 (25.2)
Hispanic or Latino	82 (15.2)
Other/mixed	74 (13.7)
Unspecified	15 (2.8)
White	213 (39.5)
ICU encounter properties	ICU encounters (N = 608)
Age at admission (%)
< 1 mo	7 (1.2)
1–23 mo	111 (18.2)
2–5 y	116 (19.1)
6–11 y	128 (21.1)
12–17 y	175 (28.8)
> 18 y	71 (11.7)
PRISb[b] score, admission (median, IQR)	9 [5–15]
PICU length of stay, d (median, IQR)	6.3 [2.1–15.9]
Hospital length of stay, d (median, IQR)	15.1 [7.3–36.3]
Mortality (%)	76 (12.5)
Associated VPS diagnoses[c] (%)
Acute respiratory failure (518.81)	447 (73.5)
Anemia (285.9)	295 (48.5)
Thrombocytopenia (287.5)	210 (34.5)
Pneumonia (486)	192 (31.6)
Impaired renal function (588.9)	182 (29.9)
Epilepsy without status epilepticus (345.9)	181 (29.8)
Glucocorticoid deficiency (255.41)	136 (22.4)
Neutropenia (288.00)	96 (15.8)
Tracheostomy status (V44.0)	95 (15.6)
Urinary tract infection (599)	92 (15.1)

Abbreviations: ICU, intensive care unit; IQR, interquartile rang; VPS, Virtual Pediatric System.

^a Categories provided by virtual pediatric system (VPS).

^b Pediatric risk of mortality (PRISM) score calculated over the first 12 hours of admission, where higher scores are associated with longer length of stay and greater risk of mortality.[38]

^c Number and percent of PICU encounters which contain this active diagnosis in the VPS database. Subsets of diagnoses were selected as most frequent and representative of the admitted population.

Fig. 1 Flowchart of PICU encounters and ADT epochs. PICU encounters were filtered by “Severe Sepsis or Septic Shock” inclusion criteria and ADT epochs were extracted from these encounters. Subsequently ADT epochs were filtered to include only those >1 hour and located in a PICU bed. Lastly, several ADT epochs contained zero physiologic streaming vital signs recorded during the epoch. ADT, admission–discharge–transfer; PICU, pediatric intensive care unit.

Completeness

The count of PSVS measurements per included ADT epoch was substantially higher (> 400-fold) compared with EVS measurements ([Table 2]). However, substantial missingness was observed across all measures from PSVS, including 125 ADT epochs (15.5% of total) which had zero PSVS recorded, compared with no epochs with zero EVS recorded. The number of ADT epochs which were completely missing PSVS measurements was significantly different by measure (HR: 126 [15.6%], RR: 131 [16.2%], SpO₂: 126 [15.6%], NBP: 136 [16.8%], ABP: 462 [57.2%]) compared with those completely missing EVS by measure (HR: 0, RR: 1 [0.1%], SpO₂: 0, NBP: 4 [0.5%], ABP: 394 [48.8%], p < 0.001) ([Fig. 2A]). Of note, although we report “missing” ABP measurements, these values are only present in patients with functioning arterial catheters and are not expected in all patients or epochs. For subsequent analyses, we filtered to remove ADT epochs which were completely missing vital signs for each measure and source. Percent missingness, filtered for those not completely missing vital signs, was significantly different between sources for all measures, analyzed by the Wilcoxon rank sum test (p < 0.001 for all except ABP, where p = 0.03).

Fig. 2 Comparison of physiologic streaming and EHR vital signs. (A) Percent of all hours in ADT epochs with missing vital sign measures, by source. (B) Violin plots with inset box-whisker plots of z-score concordance between EHR and streaming vital signs for each measure. Positive z-score indicates EHR vital sign is larger than physiologic streaming vital sign mean value, while a negative z-score indicates EHR vital sign is smaller than physiologic streaming vital sign mean value. (C) Sample time series plot of physiologic streaming vital sign (small black dots) with median filter smoothed line (blue) and EHR vital signs (large red diamonds). Shaded gray represents the time at which the physiologic streaming vital sign values are above the age-specific cutoff, shown in light blue horizontal line. No EHR vital signs shown were above age-specific cutoff. (D) Box-whisker plots of the percent of time above (or below) age-specific vital sign thresholds, per vital sign measure and source. Note that the y-axis was terminated at 80% to better highlight box differences, though whiskers and points for some measures extend beyond 80%. All measures were significantly different between sources (pair-wise Wilcox signed rank, p ≤ 0.01) and among source (Kruskal–Wallis, p < 0.001). ADT, admission–discharge–transfer; EHR, electronic health record.

Table 2 Vital sign counts by measure and source (EHR vs. physiologic streaming)

	EHR vital signs		Streaming vital signs
Measure	Total count	Median [IQR] per ADT epoch	Total count	Median [IQR] per ADT epoch
HR	315,514	174 [51–423]	129,844,418	48,610 [8,151–176,974]
SpO2	315,245	175 [51–424]	129,687,929	48,955 [8,028–176,667]
RR	297,005	162 [50–402]	109,808,648	43,732 [6,758–150,736]
NBP	94,297	54 [23–125]	38,217,594	16,238 [3,316–47,697]
ABP	82,210	2 [0–113]	38,187,564	0 [0–36,819]

Abbreviations: ABP, arterial blood pressure; EHR, electronic health record; HR, heart rate; IQR, interquartile range; NBP, noninvasive (cuff) blood pressure; RR, respiratory rate; SpO₂, peripheral oxygen saturation.

Note: Summaries per ADT epoch include only those meeting ADT epoch inclusion criteria (>1 hour and from a PICU bed, N = 808).

Concordance

Vital sign concordance was calculated across filtered ADT epochs with any PSVS (N = 677). Median calculated z-scores were positive (meaning the PSVS was larger in magnitude than the EVS) for three measures (HR: 0.11 [IQR: −0.48 to 0.61], RR: 0.11 [IQR: −0.20 to 0.44], ABP: 0.15 [IQR: −0.3 to 0.50]) and negative (meaning the EVS was larger in magnitude) for one measure (SpO₂: −0.16 [IQR: −0.58 to 0.39]). Violin plots show wider spread in z-score values for HR and narrower spread for RR ([Fig. 2B]). Scatter plots of EVS versus mean PSVS demonstrate generally good agreement for HR (r ² = 0.95, root mean square error [RMSE]: 0.08) but less good agreement for the remaining measures (SpO₂: r ² = 0.74, RMSE = 0.04; RR: r ² = 0.77, RMSE = 1.43; ABP: r ² = 0.68, RMSE = 0.57) ([Fig. 3]). Bland–Altman plots confirm negative bias (mean difference, EHR minus streaming) suggesting larger magnitude PSVS compared with EVS for the same three measures (HR: mean −0.29, SD 6.2; RR: mean −1.19, SD 6.1; ABP: mean −0.76, SD 9.9) and positive bias for one measure (SpO₂: mean 0.19, SD 2.0) ([Supplementary Fig. S1] [available in the online version]).

Fig. 3 Scatter plots of physiologic streaming versus EHR-recorded vital sign concordance by measure. Each point represents the mean of the physiologic streaming vital signs from the ± 20 minutes surrounding each EHR-recorded vital sign. Contours (red) are overlaid to highlight the distribution of overlapping points. ABP, arterial blood pressure; HR, heart rate; RR, respiratory rate; SpO₂, peripheral oxygen saturation.

Plausibility

Time above (or below) vital sign cutoffs was calculated for each filtered ADT epoch with PSVS or EVS (N = 678) and normalized to the total time each measure was available, as this time was different by measure for each epoch (see the Completeness section above and [Fig. 2C]). Percent of time above or below cutoffs was significantly different by source for each measure, as well as significantly different by measure ([Fig. 2D]). SpO₂ and ABP demonstrated the smallest percent of time below cutoff (SpO₂: EHR 0.9% [0.4–2.3] vs. streaming 3.5% [1.2–7.7], p < 0.001; SBP: EHR 3.3% [0.8–9.6] vs. streaming 5.9% [1.5–18.0], p < 0.001), while HR and RR demonstrated the largest percent of time above age-specific cutoffs (HR: EHR 10.7% [3.3–27.6] vs. streaming 13.5% [3.5–35.7], p = 0.01; RR: EHR 23.0% [10.1–40.5] vs. streaming 34.9% [16.3–58.7], p < 0.001).

Currency

All measures demonstrated a positive time difference between entry time and recorded time, indicating a delay in entering these values into the EHR. The magnitude of delay, in minutes, was significantly different by measure (Kruskal–Wallis rank sum, p < 0.001) and posthoc pairwise testing demonstrated significant differences in time difference among all pairs except for SpO₂ (6 minutes [1–22]) and HR (6 minutes [1–22]). The greatest median time difference was NBP (9 minutes, [1–27]) and least median times were SpO₂ and HR. All measures also included values with negative time differences, indicating the values were entered into the EHR prior to the documented recorded time from the patient or that the documented recorded time was inaccurate, with the greatest percentage in the NBP (15.1%) and the lowest percent from the ABP (7.7%). Density distribution of time difference for representative measure (HR) is shown in [Supplementary Fig. S2] (available in the online version).

Discussion

In this study, we identified significant differences between vital sign sources across physiologic measures, suggesting that each source has advantages and disadvantages that must be considered when used for prospective or retrospective analyses. Concordance was highest among HR measurements and lowest among BP measurements, though bias existed across all measures. Plausibility significantly differed among measures and is critical for vital sign use in either prospective decision support or retrospective studies. All EVS measures demonstrated a mean delay from time recorded at the patient to EHR entry. Although these results may be expected, to our knowledge this is the first study to systematically compare these measures across data domains in critically ill children.

In our institution, PSVSs are captured at the bedside and displayed on monitors, both in the patient's room and at a central unit location. Monitor alarms are triggered from these measures based on parameters established by clinicians and nurses, and typically transmit to nursing communication devices. Parameterized vital signs stream to our EHR regularly and are edited or verified by nurses, who may also document independent of these suggested vital signs. Based on this flow into the EHR, we hypothesized that EVS would be highly concordant with PSVS. Although measures were similar, we did identify bias in each measure. These biases may reflect “smoothing,” as described by others in adult patients,[32] [33] where manually charted data were less likely to reflect abnormal values. Interestingly, the direction of these differences varied by measure, with EHR-recorded SpO₂ tending to raise the value while HR, RR, and ABP tended to lower the value. This is perhaps related to measurement artifact on SpO₂ waveforms which tend to artifactually lower SpO₂, or similar ECG-derived respiration measures which can artifactually raise HR and RR (e.g., some forms of chest physiotherapy can artifactually mimic ventricular tachycardia on three-lead ECG measurements). Agreement was best (highest r ² value) for HR, which indicates it may be the measure least affected by artifact. Because deterioration indices or other decision support relies on vital signs to trigger warnings about patients in the intensive care unit (ICU), developers must understand each data source's own strengths, limitations, and biases.[33]

In this population of sepsis patients, we chose to examine plausibility not simply on whether or not a value is likely to be true, but rather based on the vital sign's relationship to predefined age-adjusted sepsis vital sign criteria. This definition was chosen to mimic detection of “abnormal” vital signs by a decision support system, one anticipated use of these vital sign sources. We hypothesized that this cohort would be enriched for values which suggest patients meet these rudimentary sepsis vital sign criteria. Although concordance was similar, plausibility percentages were different between sources across all vital sign measures. These differences could be explained by the more granular nature of PSVS, whereby patients' vital signs may ebb and flow within an hour between EVS (which may demonstrate relative concordance in the time surrounding that recorded vital sign). This ebb and flow is critical to both illness detection such as sepsis surveillance[8] and to retrospective studies using “time above” or “time below” a priori cutoff as exposure variables. For example, the use of granular PSVS to measure hypotension as opposed to EVS may provide stronger evidence for minimizing systolic hypotension in post-arrest care.[34] In the future, even more granular data sources, such as continuous waveform data, may further improve illness detection but may require advanced analysis techniques such as machine learning. It may not be surprising that HR and RR were the measures which most often fell above the vital sign cutoff in this patient population. This may be due to elevated HR in sepsis patients. Additionally, the chosen cutoffs, while age-based, were clustered into only five separate age groups. As discussed above, ECG-derived respiration may over-estimate compared with charted RR and may contribute to the increase in percent above plausible values for this measure.

Previous studies have demonstrated errors in manual documentation of vital signs from a variety of reasons.[14] [15] [35] Kallioinen et al described five sources of inaccuracy related to adult RR measurement including awareness effect (similar to the Hawthorne effect), observation methods, observer variability, value bias, and recording omission.[15] Similarly, Skyttberg et al identified low completeness and poor currency in EHR-documented emergency department vital signs and concluded that these were insufficient for early warning system usage.[14] Our results also demonstrate deficiencies in completeness, though these were predominantly in PSVS and likely due to technical glitches (see below); EVSs were nearly perfectly complete with the exception of BP measures which were dependent on invasive monitoring placement.

This study confirms the poor currency of EVS suggested in other studies, with a median 6- to 9-minute delay though with the upper quartile experiencing >20-minute delay in documentation. Skyttberg et al[36] have examined factors related to these currency delays and proposed a series of steps for vital sign quality improvement in the emergency room, some of which are applicable to the ICU environment (e.g., provide workflow support and perform quality control). Ultimately, a delay in data entry presents challenges for the delivery of real-time CDS in an acute care environment.

In addition to the limitation of this being a single-center study, we observed technical limitations which caused substantial missingness in our PSVS measures. Unfortunately, because no business operations systems were making use of the streaming parameterized vital signs, long periods of data missingness went unnoticed and resulted in gaps in stored data. Reviews of these disruptions suggested that buffering capabilities in the messaging interface was responsible for the initial failure, followed by lack of “monitoring of monitors” systems to notify the appropriate supports. These failures have been mitigated in the most recent 6 months' worth of PSVS data. Fortunately, the missingness was random with respect to patients, rooms, and time of year, so should not have systematically biased these results. These challenges suggest, similar to automated anomaly detection recommended in CDS implementations,[37] that institutions should plan automated streaming monitor systems when using PSVS in production clinical environments.

Conclusion

Vital signs are critical features of prospective CDS systems and covariates in retrospective analyses. Both PSVS and EVS have advantages and disadvantages, and the trade-off between decreasing bias and increasing currency must be weighed against the possibility of artifactual plausibility issues and technical limitations related to completeness. Real-time CDS implementations should consider cross-domain retrospective analyses and comparisons across data sources and measures, in addition to accounting for source anomaly detection.

Clinical Relevance Statement

Vital signs provide critical objective data for patient care in the intensive care unit and beyond, and are also features in clinical decision support tools such as early warning systems or best-practice alerts. Accurate, timely, and plausible representation of patient vital signs is necessary for these tools to operate appropriately. In this study, we demonstrate the advantages and disadvantages of physiologic streaming and EHR-recorded vital signs.

Conflict of Interest

None declared.

Acknowledgments

The authors would like to thank Dr. Akira Nishisaki, Danielle Traynor, and the CHOP Center for Critical Care Effectiveness and Outcomes (CCCEO) for their assistance with the Virtual Pediatric System (VPS) quality improvement data. Additionally, they would like to thank the CHOP Information Systems and Research Information Systems teams, including Fernando Gonzalez, Jim Gay, Jay Jenkins, and Mohammad Saeed, for their assistance with acquisition, troubleshooting, and storage of physiologic streaming vital signs.

Protection of Human and Animal Subjects

This study was granted an exemption (# 19–016133, initial approval 5/24/2019, addendum approval 12/16/2020) by the Institutional Review Board at the Children's Hospital of Philadelphia.

Institution where this work was performed: The Children's Hospital of Philadelphia.

• References

• 1 Watson RS, Carcillo JA. Scope and epidemiology of pediatric sepsis. Pediatr Crit Care Med 2005; 6 (3, Suppl): S3-S5
• 2 Weiss SL, Fitzgerald JC, Pappachan J. et al; Sepsis Prevalence, Outcomes, and Therapies (SPROUT) Study Investigators and Pediatric Acute Lung Injury and Sepsis Investigators (PALISI) Network. Global epidemiology of pediatric severe sepsis: the sepsis prevalence, outcomes, and therapies study. Am J Respir Crit Care Med 2015; 191 (10) 1147-1157
• 3 Weiss SL, Fitzgerald JC, Balamuth F. et al. Delayed antimicrobial therapy increases mortality and organ dysfunction duration in pediatric sepsis. Crit Care Med 2014; 42 (11) 2409-2417
• 4 Reyna MA, Josef CS, Jeter R. et al. Early prediction of sepsis from clinical data: the PhysioNet/Computing in Cardiology Challenge 2019. Crit Care Med 2020; 48 (02) 210-217
• 5 Masino AJ, Harris MC, Forsyth D. et al. Machine learning models for early sepsis recognition in the neonatal intensive care unit using readily available electronic health record data. PLoS One 2019; 14 (02) e0212665
• 6 Nemati S, Holder A, Razmi F, Stanley MD, Clifford GD, Buchman TG. An interpretable machine learning model for accurate prediction of sepsis in the ICU. Crit Care Med 2018; 46 (04) 547-553
• 7 Umscheid CA, Betesh J, VanZandbergen C. et al. Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med 2015; 10 (01) 26-31
• 8 Hsu HE, Abanyie F, Agus MSD. et al. A national approach to pediatric sepsis surveillance. Pediatrics 2019; 144 (06) e20191790
• 9 Eisenberg M, Madden K, Christianson JR, Melendez E, Harper MB. Performance of an automated screening algorithm for early detection of pediatric severe sepsis. Pediatr Crit Care Med 2019; 20 (12) e516-e523
• 10 Weiss SL, Balamuth F, Chilutti M. et al. Identification of pediatric sepsis for epidemiologic surveillance using electronic clinical data. Pediatr Crit Care Med 2020; 21 (02) 113-121
• 11 Martin B, Bennett TD. Sepsis computable phenotypes in the service of observational research. Crit Care Med 2019; 47 (02) 303-305
• 12 Wager KA, Schaffner MJ, Foulois B, Swanson Kazley A, Parker C, Walo H. Comparison of the quality and timeliness of vital signs data using three different data-entry devices. Comput Inform Nurs 2010; 28 (04) 205-212
• 13 Stevenson JE, Israelsson J, Nilsson GC, Petersson GI, Bath PA. Recording signs of deterioration in acute patients: the documentation of vital signs within electronic health records in patients who suffered in-hospital cardiac arrest. Health Informatics J 2016; 22 (01) 21-33
• 14 Skyttberg N, Chen R, Blomqvist H, Koch S. Exploring vital sign data quality in electronic health records with focus on emergency care warning scores. Appl Clin Inform 2017; 8 (03) 880-892
• 15 Kallioinen N, Hill A, Christofidis MJ, Horswill MS, Watson MO. Quantitative systematic review: sources of inaccuracy in manually measured adult respiratory rate data. J Adv Nurs 2021; 77 (01) 98-124
• 16 De Georgia MA, Kaffashi F, Jacono FJ, Loparo KA. Information technology in critical care: review of monitoring and data acquisition systems for patient care and research. Scientific World Journal 2015; 2015: 727694
• 17 Goodwin AJ, Eytan D, Greer RW. et al. A practical approach to storage and retrieval of high-frequency physiological signals. Physiol Meas 2020; 41 (03) 035008
• 18 Matam BR, Duncan H. Technical challenges related to implementation of a formula one real time data acquisition and analysis system in a paediatric intensive care unit. J Clin Monit Comput 2018; 32 (03) 559-569
• 19 Ghassemi M, Celi LA, Stone DJ. State of the art review: the data revolution in critical care. Crit Care 2015; 19: 118
• 20 Sun Y, Guo F, Kaffashi F, Jacono FJ, DeGeorgia M, Loparo KA. INSMA: an integrated system for multimodal data acquisition and analysis in the intensive care unit. J Biomed Inform 2020; 106: 103434
• 21 Lindell RB, Nishisaki A, Weiss SL. et al. Comparison of methods for identification of pediatric severe sepsis and septic shock in the virtual pediatric systems database. Crit Care Med 2019; 47 (02) e129-e135
• 22 Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc 2013; 20 (01) 144-151
• 23 Chiotos K, Fitzgerald J, Weiss S. et al. Suspected sepsis clinical pathway. Published 2019. Accessed February 20, 2020 at: https://www.chop.edu/clinical-pathway/sepsis-inpatient-care-clinical-pathway
• 24 Goldstein B, Giroir B, Randolph A. International Consensus Conference on Pediatric Sepsis. International pediatric sepsis consensus conference: definitions for sepsis and organ dysfunction in pediatrics. Pediatr Crit Care Med 2005; 6 (01) 2-8
• 25 Team Rs. RStudio: Integrated Development Environment for R. Published online 2016. Accessed July 3, 2022 at: http://www.rstudio.com/
• 26 Hester J, Wickham H. odbc: Connect to ODBC Compatible Databases (using the DBI Interface). Published online 2020. Accessed July 3, 2022 at: https://cran.r-project.org/package=odbc
• 27 Wickham H, Ruiz E. dbplyr: A “dplyr” Back End for Databases. Published online 2020. Accessed July 3, 2022 at: https://cran.r-project.org/package=dbplyr
• 28 Wickham H, Francois R, Henry L, Müller K. dplyr: A Grammar of Data Manipulation. Published online 2017. Accessed July 3, 2022 at: https://cran.r-project.org/package=dplyr
• 29 Wickham H. Ggplot2: Elegant Graphics for Data Analysis. New York, NY: Springer-Verlag; 2016
• 30 Wilke CO. cowplot: Streamlined Plot Theme and Plot Annotations for “ggplot2.” Published online 2017. Accessed July 3, 2022 at: https://cran.r-project.org/package=cowplot
• 31 Xie Y. knitr: A General-Purpose Package for Dynamic Report Generation in R. Published online 2020. Accessed July 3, 2022 at: https://yihui.org/knitr/
• 32 Sapo M, Wu S, Asgari S. et al. A comparison of vital signs charted by nurses with automated acquired values using waveform quality indices. J Clin Monit Comput 2009; 23 (05) 263-271
• 33 Feldman K, Rohan AJ, Chawla NV. Discrete heart rate values or continuous streams? Representation, variability, and meaningful use of vital sign data. Comput Inform Nurs 2021; 39 (11) 793-803
• 34 Laverriere EK, Polansky M, French B, Nadkarni VM, Berg RA, Topjian AA. Association of duration of hypotension with survival after pediatric cardiac arrest. Pediatr Crit Care Med 2020; 21 (02) 143-149
• 35 Maslove DM, Dubin JA, Shrivats A, Lee J. Errors, omissions, and outliers in hourly vital signs measurements in intensive care. Crit Care Med 2016; 44 (11) e1021-e1030
• 36 Skyttberg N, Vicente J, Chen R, Blomqvist H, Koch S. How to improve vital sign data quality for use in clinical decision support systems? A qualitative study in nine Swedish emergency departments. BMC Med Inform Decis Mak 2016; 16: 61
• 37 Orenstein EW, Muthu N, Weitkamp AO. et al. Towards a maturity model for clinical decision support operations. Appl Clin Inform 2019; 10 (05) 810-819
• 38 Pollack MM, Patel KM, Ruttimann UE. PRISM III: an updated pediatric risk of mortality score. Crit Care Med 1996; 24 (05) 743-752

Publication History

Received: 22 April 2021

Accepted: 15 June 2022

Article published online:
18 September 2022