Validation of test performance and clinical time zero for an electronic health record embedded severe sepsis alert

Summary

Bachground

Increasing use of EHRs has generated interest in the potential of computerized clinical decision support to improve treatment of sepsis. Electronic sepsis alerts have had mixed results due to poor test characteristics, the inability to detect sepsis in a timely fashion and the use of outside software limiting widespread adoption. We describe the development, evaluation and validation of an accurate and timely severe sepsis alert with the potential to impact sepsis management.

Objective

To develop, evaluate, and validate an accurate and timely severe sepsis alert embedded in a commercial EHR.

Methods

he sepsis alert was developed by identifying the most common severe sepsis criteria among a cohort of patients with ICD 9 codes indicating a diagnosis of sepsis. This alert requires criteria in three categories: indicators of a systemic inflammatory response, evidence of suspected infection from physician orders, and markers of organ dysfunction. Chart review was used to evaluate test performance and the ability to detect clinical time zero, the point in time when a patient develops severe sepsis.

Results

Two physicians reviewed 100 positive cases and 75 negative cases. Based on this review, sensitivity was 74.5%, specificity was 86.0%, the positive predictive value was 50.3%, and the negative predictive value was 94.7%. The most common source of end-organ dysfunction was MAP less than 70 mm/Hg (59%). The alert was triggered at clinical time zero in 41% of cases and within three hours in 53.6% of cases. 96% of alerts triggered before a manual nurse screen.

Conclusion

We are the first to report the time between a sepsis alert and physician chart-review clinical time zero. Incorporating physician orders in the alert criteria improves specificity while maintaining sensitivity, which is important to reduce alert fatigue. By leveraging standard EHR functionality, this alert could be implemented by other healthcare systems.

Keywords

Testing and evaluation - inpatient care - medicine - clinical decision support - performance improvement

 

Conflicts of Interest

The authors declare that they have no conflicts of interest in the research

• References

• 1 Gaieski DF, Edwards JM, Kallan MJ, Carr BG. Benchmarking the incidence and mortality of severe sepsis in the United States. Crit Care Med 2013; 41 (05) 1167-1174.
• 2 Martin GS. Sepsis, severe sepsis and septic shock: changes in incidence, pathogens and outcomes. Expert Rev Anti Infect Ther 2012; 10 (06) 701-706.
• 3 Gaieski DF, Mikkelsen ME, Band RA, Pines JM, Massone R, Furia FF, Shofer FS, Goyal M. Impact of time to antibiotics on survival in patients with severe sepsis or septic shock in whom early goal-directed therapy was initiated in the emergency department. Crit Care Med 2010; 38 (04) 1045-1053.
• 4 Puskarich MA, Trzeciak S, Shapiro NI, Arnold RC, Horton JM, Studnek JR, Kline JA, Jones AE. Emergency Medicine Shock Research Network (EMSHOCKNET). Association between timing of antibiotic administration and mortality from septic shock in patients treated with a quantitative resuscitation protocol. Crit Care Med 2011; 39 (09) 2066-2071.
• 5 Levy MM, Dellinger RP, Townsend SR, Linde-Zwirble WT, Marshall JC, Bion J, Schorr C, Artigas A, Ramsay G, Beale R, Parker MM, Gerlach H, Reinhart K, Silva E, Harvey M, Regan S, Angus DC. The Surviving Sepsis Campaign: results of an international guideline-based performance improvement program targeting severe sepsis. Intensive Care Med 2010; 36 (02) 222-231.
• 6 Stoneking L, Denninghoff K, Deluca L, Keim SM, Munger B. Sepsis bundles and compliance with clinical guidelines. J Intensive Care Med 2011; 26 (03) 172-182.
• 7 Umscheid CA, Betesh J, VanZandbergen C, Hanish A, Tait G, Mikkelsen ME, French B, Fuchs BD. Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med 2015; 10 (01) 26-31.
• 8 Evans RS, Kuttler KG, Simpson KJ, Howe S, Crossno PF, Johnson KV, Schreiner MN, Lloyd JF, Tettelbach WH, Keddington RK, Tanner A, Wilde C, Clemmer TP. Automated detection of physiologic deterioration in hospitalized patients. J Am Med Inform Assoc JAMIA 2015; 22 (02) 350-360.
• 9 Nguyen SQ, Mwakalindile E, Booth JS, Hogan V, Morgan J, Prickett CT, Donnelly JP, Wang HE. Automated electronic medical record sepsis detection in the emergency department. PeerJ 2014; 02: e343.
• 10 Amland RC, Hahn-Cover KE. Clinical Decision Support for Early Recognition of Sepsis. Am J Med Qual 2016; 31 (02) 103-110.
• 11 Nelson JL, Smith BL, Jared JD, Younger JG. Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med 2011; 57 (05) 500-504.
• 12 Sawyer AM, Deal EN, Labelle AJ, Witt C, Thiel SW, Heard K, Reichley RM, Micek ST, Kollef MH. Implementation of a real-time computerized sepsis alert in nonintensive care unit patients. Crit Care Med 2011; 39 (03) 469-473.
• 13 Hooper MH, Weavind L, Wheeler AP, Martin JB, Gowda SS, Semler MW, Hayes RM, Albert DW, Deane NB, Nian H, Mathe JL, Nadas A, Sztipanovits J, Miller A, Bernard GR, Rice TW. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit^* . Crit Care Med 2012; 40 (07) 2096-2101.
• 14 McRee L, Thanavaro JL, Moore K, Goldsmith M, Pasvogel A. The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung J Crit Care 2014; 43 (06) 546-549.
• 15 Cruz AT, Williams EA, Graf JM, Perry AM, Harbin DE, Wuestner ER, Patel B. Test characteristics of an automated age- and temperature-adjusted tachycardia alert in pediatric septic shock. Pediatr Emerg Care 2012; 28 (09) 889-894.
• 16 Gerald D, Alsip J, Hicks J, Waldrum M, Dunlap N. Using the emr to perform continuous, automated, real-time surveillance to identify hospitalized patients at risk of sepsis. Chest 2011; 01 (140 (Meeting Abstracts)) 426A.
• 17 Makam AN, Nguyen OK, Auerbach AD. Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: A systematic review. J Hosp Med 2015; 10 (06) 396-402.
• 18 Alsolamy S, Al Salamah M, Al Thagafi M, Al-Dorzi HM, Marini AM, Aljerian N, Al-Enezi F, Al-Hunaidi E, Mahmoud AM, Alamry A, Arabi YM. Diagnostic accuracy of a screening electronic alert tool for severe sepsis and septic shock in the emergency department. BMC Med Inform Decis Mak 2014; 14: 105.
• 19 Meurer WJ, Smith BL, Losman ED, Sherman D, Yaksich JD, Jared JD, Malani FN, Younger JD. Real-time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc 2009; 57 (01) 40-45.
• 20 Narayanan N, Gross AK, Pintens M, Fee C, MacDougall C. Effect of an electronic medical record alert for severe sepsis among ED patients. Am J Emerg Med 2016; 34 (02) 185-188.
• 21 Statement from SSC Leadership on Time Zero in the Emergency Department [Internet], Surviving Sepsis Campaign. [cited 2016 Jan 20]; Available from: http://www.survivingsepsis.org/SiteCollectionDocuments/Time-Zero.pdf.
• 22 Jayanthi A. Top 10 EHR vendors by overall market share. Becker’s Health IT & CIO Review. 2015 Feb 13. Available from: http://www.beckershospitalreview.com/healthcare-information-technology/top-10-ehr-vendors-by-overall-market-share.html.
• 23 Iwashyna TJ, Odden A, Rohde J, Bonham C, Kuhn L, Malani P, Chen L, Flanders S. Identifying patients with severe sepsis using administrative claims: patient-level validation of the angus implementation of the international consensus conference definition of severe sepsis. Med Care 2014; 52 (06) e39-e43.
• 24 Dellinger RP, Levy MM, Rhodes A, Annane D, Gerlach H, Opal SM, Sevransky JE, Sprung CL, Douglas IS, Jaeschke R, Osborn TM, Nunnally ME, Townsend SR, Reinhart K, Kleinpell RM, Angus DC, Deutschman CS, Machado FR, Rubenfeld GD, Webb SA, Beale RJ, Vincent JL, Moreno R. Surviving Sepsis Campaign Guidelines Committee including the Pediatric Subgroup. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med 2013; 41 (02) 580-637.
• 25 Brandt BN, Gartner AB, Moncure M, Cannon CM, Carlton E, Cleek C, Wittkopp C, Simpson SQ. Identifying Severe Sepsis via Electronic Surveillance. Am J Med Qual Off J Am Coll Med Qual 2015; 30 (06) 559-565.
• 26 Delate T, Bowles EJ, Pardee R, Wellman RD, Habel LA, Yood MU, Nekhlyudov L, Goddard KA, Davis RL, McCarty CA, Onitilo AA, Feigleson HS, Freml J, Wagner E. Validity of eight integrated healthcare delivery organizations’ administrative clinical data to capture breast cancer chemotherapy exposure. Cancer Epidemiol Biomark Prev Publ Am Assoc Cancer Res Cosponsored Am Soc Prev Oncol 2012; 21 (04) 673-680.
• 27 Stein BD, Bautista A, Schumock GT, Lee TA, Charbeneau JT, Lauderdale DS, Naureckas ET, Meltzer DO, Krishnan JA. The validity of International Classification of Diseases, Ninth Revision, Clinical Modification diagnosis codes for identifying patients hospitalized for COPD exacerbations. Chest 2012; 141 (01) 87-93.
• 28 StataCorp. 2013. Stata Statistical Software: Release 13. College Station. TX: StataCorp LP.;
• 29 Whittaker S-A, Mikkelsen ME, Gaieski DF, Koshy S, Kean C, Fuchs BD. Severe sepsis cohorts derived from claims-based strategies appear to be biased toward a more severely ill patient population. Crit Care Med 2013; Apr; 41 (04) 945-53.
• 30 Demner-Fushman D, Chapman WW, McDonald CJ. What can natural language processing do for clinical decision support?. J Biomed Inform 2009; 42 (05) 760-772.