Effect of a Real-Time Risk Score on 30-day Readmission Reduction in Singapore

Abstract

Objective To develop a risk score for the real-time prediction of readmissions for patients using patient specific information captured in electronic medical records (EMR) in Singapore to enable the prospective identification of high-risk patients for enrolment in timely interventions.

Methods Machine-learning models were built to estimate the probability of a patient being readmitted within 30 days of discharge. EMR of 25,472 patients discharged from the medicine department at Ng Teng Fong General Hospital between January 2016 and December 2016 were extracted retrospectively for training and internal validation of the models. We developed and implemented a real-time 30-day readmission risk score generation in the EMR system, which enabled the flagging of high-risk patients to care providers in the hospital. Based on the daily high-risk patient list, the various interfaces and flow sheets in the EMR were configured according to the information needs of the various stakeholders such as the inpatient medical, nursing, case management, emergency department, and postdischarge care teams.

Results Overall, the machine-learning models achieved good performance with area under the receiver operating characteristic ranging from 0.77 to 0.81. The models were used to proactively identify and attend to patients who are at risk of readmission before an actual readmission occurs. This approach successfully reduced the 30-day readmission rate for patients admitted to the medicine department from 11.7% in 2017 to 10.1% in 2019 (p < 0.01) after risk adjustment.

Conclusion Machine-learning models can be deployed in the EMR system to provide real-time forecasts for a more comprehensive outlook in the aspects of decision-making and care provision.

Keywords

machine learning - data modeling - implementation and deployment - testing and evaluation - electronic health records and systems

Author Contributions

C.X.W. provided the study concept and design and contributed to the critical review of the statistical analysis and manuscript revision. E.S. participated in the conception and design of the interventions and contributed to the manuscript revision. F.W.L.P. constructed the write-up of the manuscript and participated in the statistical analysis. K.P.T. built the risk score model and performed the statistical analysis. J.P. participated in constructing the intervention write-up and reviewed the manuscript for important intellectual content. J.L.A.D.S. contributed in building the risk score into the NTFGH Epic EMR system. W.S.T. provided advice to the team on the initial model development and reviewed the manuscript for important intellectual content. P.P. reviewed the manuscript for important intellectual content. K.S.M.L. provided inputs on the intervention write-up. G.Y.H.T. oversaw the implementation of the risk score into the hospital's EMR system. G.S.W.C. oversaw the conception and design of the interventions and provided advice on parameters with clinical significance. C.H.H. contributed to the critical review of the study design, statistical analysis, and reviewed the manuscript for important intellectual content. All the authors reviewed and approved the final manuscript.

Note

This study received ethical approval from the National Healthcare Group Domain Specific Review Board (NHG DSRB 2021/00284).

Publication History

Received: 29 October 2020

Accepted: 12 May 2021

Article published online:
19 May 2021

• References

• 1 Kripalani S, Theobald CN, Anctil B, Vasilevskis EE. Reducing hospital readmission rates: current strategies and future directions. Annu Rev Med 2014; 65 (01) 471-485
• 2 Hines AL. (Truven Health Analytics), Barrett ML (ML Barrett, Inc), Jiang HJ (AHRQ), and Steiner CA (AHRQ), Conditions with the Largest Number of Adult Hospital Readmissions by Payer, 2011. HCUP Statistical Brief #172. April 2014. Rockville, MD: Agency for Healthcare Research and Quality;
• 3 Felix HC, Seaberg B, Bursac Z, Thostenson J, Stewart MK. Why do patients keep coming back? Results of a readmitted patient survey. Soc Work Health Care 2015; 54 (01) 1-15
• 4 Longman JMI, Rolfe MI, Passey MD. et al. Frequent hospital admission of older people with chronic disease: a cross-sectional survey with telephone follow-up and data linkage. BMC Health Serv Res 2012; 12: 373
• 5 Tuso P, Huynh DN, Garofalo L. et al. The readmission reduction program of Kaiser Permanente Southern California-knowledge transfer and performance improvement. Perm J 2013; 17 (03) 58-63
• 6 Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff (Millwood) 2014; 33 (07) 1123-1131
• 7 Kocher RP, Adashi EY. Hospital readmissions and the Affordable Care Act: paying for coordinated quality care. JAMA 2011; 306 (16) 1794-1795
• 8 van Walraven C, Bennett C, Jennings A, Austin PC, Forster AJ. Proportion of hospital readmissions deemed avoidable: a systematic review. CMAJ 2011; 183 (07) E391-E402
• 9 Donzé J, Aujesky D, Williams D, Schnipper JL. Potentially avoidable 30-day hospital readmissions in medical patients: derivation and validation of a prediction model. JAMA Intern Med 2013; 173 (08) 632-638
• 10 Oddone EZ, Weinberger M, Horner M. et al. Classifying general medicine readmissions. Are they preventable? Veterans Affairs Cooperative Studies in Health Services Group on Primary Care and Hospital Readmissions. J Gen Intern Med 1996; 11 (10) 597-607
• 11 McInnes EG, Joshi DM, O'Brien TD. Failed discharges: setting standards for improvement. Geriatr Med 1988; 18: 35-42
• 12 Balla U, Malnick S, Schattner A. Early readmissions to the department of medicine as a screening tool for monitoring quality of care problems. Medicine (Baltimore) 2008; 87 (05) 294-300
• 13 Gruneir A, Dhalla IA, van Walraven C. et al. Unplanned readmissions after hospital discharge among patients identified as being at high risk for readmission using a validated predictive algorithm. Open Med 2011; 5 (02) e104-e111
• 14 Vest JR, Gamm LD, Oxford BA, Gonzalez MI, Slawson KM. Determinants of preventable readmissions in the United States: a systematic review. Implement Sci 2010; 5: 88
• 15 Costantino ME, Frey B, Hall B, Painter P. The influence of a postdischarge intervention on reducing hospital readmissions in a Medicare population. Popul Health Manag 2013; 16 (05) 310-316
• 16 Low LL, Tay WY, Ng MJ, Tan SY, Liu N, Lee KH. Frequent hospital admissions in Singapore: clinical risk factors and impact of socioeconomic status. Singapore Med J 2018; 59 (01) 39-43
• 17 Leppin AL, Gionfriddo MR, Kessler M. et al. Preventing 30-day hospital readmissions: a systematic review and meta-analysis of randomized trials. JAMA Intern Med 2014; 174 (07) 1095-1107
• 18 Hansen LO, Young RS, Hinami K, Leung A, Williams MV. Interventions to reduce 30-day rehospitalization: a systematic review. Ann Intern Med 2011; 155 (08) 520-528
• 19 Dash S, Shakyawar SK, Sharma M. et al. Big data in healthcare: management, analysis and future prospects. J Big Data 2019; 6: 54
• 20 McKinsey Global Institute, (Dec 2016). The age of Analytics: Competing in a Data-Driven World. Accessed May 2020 at: https://www.mckinsey.com/business-functions/mckinsey-analytics/our-insights/the-age-of-analytics-competing-in-a-data-driven-world
• 21 Futoma J, Morris J, Lucas J. A comparison of models for predicting early hospital readmissions. J Biomed Inform 2015; 56: 229-238
• 22 Frizzell JD, Liang L, Schulte PJ. et al. Prediction of 30-day all-cause readmissions in patients hospitalized for heart failure: comparison of machine learning and other statistical approaches. JAMA Cardiol 2017; 2 (02) 204-209
• 23 Mortazavi BJ, Downing NS, Bucholz EM. et al. Analysis of machine learning techniques for heart failure readmissions. Circ Cardiovasc Qual Outcomes 2016; 9 (06) 629-640
• 24 Billings J, Blunt I, Steventon A, Georghiou T, Lewis G, Bardsley M. Development of a predictive model to identify inpatients at risk of re-admission within 30 days of discharge (PARR-30). BMJ Open 2012; 2 (04) e001667
• 25 Zhou H, Della PR, Roberts P, Goh L, Dhaliwal SS. Utility of models to predict 28-day or 30-day unplanned hospital readmissions: an updated systematic review. BMJ Open 2016; 6 (06) e011060
• 26 Amarasingham R, Moore BJ, Tabak YP. et al. An automated model to identify heart failure patients at risk for 30-day readmission or death using electronic medical record data. Med Care 2010; 48 (11) 981-988
• 27 Cholleti S, Post A, Gao J. et al. Leveraging derived data elements in data analytic models for understanding and predicting hospital readmissions. AMIA Annu Symp Proc 2012; 2012: 103-111
• 28 Dagliati A, Tibollo V, Sacchi L. Big data as a driver for clinical decision support systems: a learning health systems perspective. Front Digit Humanit 2018. Doi: 3389/fdigh.2018.00008
• 29 Romero-Brufau S, Wyatt KD, Boyum P, Mickelson M, Moore M, Cognetta-Rieke C. Implementation of artificial intelligence-based clinical decision support to reduce hospital readmissions at a regional hospital. Appl Clin Inform 2020; 11 (04) 570-577
• 30 Kansagara D, Englander H, Salanitro A. et al. Risk prediction models for hospital readmission: a systematic review. JAMA 2011; 306 (15) 1688-1698
• 31 King's Fund, Predictive Risk Project [Internet]. King's Fund. 2005 . Accessed May 2020 at: http://www.kingsfund.org.uk/sites/files/kf/field/field_document/predictive-risk-literature-review-june2005.pdf
• 32 Tsui E, Au SY, Wong CP, Cheung A, Lam P. Development of an automated model to predict the risk of elderly emergency medical admissions within a month following an index hospital visit: a Hong Kong experience. Health Informatics J 2015; 21 (01) 46-56
• 33 van Walraven C, Dhalla IA, Bell C. et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ 2010; 182 (06) 551-557
• 34 Canadian Institute for Health Information, Early identification of people at-risk of hospitalization [internet]. Ontario: Queen's Printer for Ontario. 2013 . Accessed May 2020 at: http://www.hqontario.ca/Portals/0/Documents/qi/harp-a-new-tool-for-supporting-providers-and-patients.pdf
• 35 Eckert C, Nieves-Robbins N, Spieker E. et al. Development and prospective validation of a machine learning-based risk of readmission model in a large military hospital. Appl Clin Inform 2019; 10 (02) 316-325
• 36 Zhang Y, Zhang Y, Sholle E. et al. Assessing the impact of social determinants of health on predictive models for potentially avoidable 30-day readmission or death. PLoS One 2020; 15 (06) e0235064
• 37 Bernheim SM, Spertus JA, Reid KJ. et al. Socioeconomic disparities in outcomes after acute myocardial infarction. Am Heart J 2007; 153 (02) 313-319
• 38 Calvillo-King L, Arnold D, Eubank KJ. et al. Impact of social factors on risk of readmission or mortality in pneumonia and heart failure: systematic review. J Gen Intern Med 2013; 28 (02) 269-282
• 39 Mahmoudi E, Kamdar N, Kim N, Gonzales G, Singh K, Waljee AK. Use of electronic medical records in development and validation of risk prediction models of hospital readmission: systematic review. BMJ 2020; 369: m958
• 40 Krumholz HM, Bernheim SM. Considering the role of socioeconomic status in hospital outcomes measures. Ann Intern Med 2014; 161 (11) 833-834
• 41 Hu J, Gonsahn MD, Nerenz DR. Socioeconomic status and readmissions: evidence from an urban teaching hospital. Health Aff (Millwood) 2014; 33 (05) 778-785
• 42 Filc D, Davidovich N, Novack L, Balicer RD. Is socioeconomic status associated with utilization of health care services in a single-payer universal health care system?. Int J Equity Health 2014; 13 (01) 115
• 43 Nagasako EM, Reidhead M, Waterman B, Dunagan WC. Adding socioeconomic data to hospital readmissions calculations may produce more useful results. Health Aff (Millwood) 2014; 33 (05) 786-791
• 44 Feldman SS, Davlyatov G, Hall AG. Toward understanding the value of missing social determinants of health data in care transition planning. Appl Clin Inform 2020; 11 (04) 556-563
• 45 Tan SY, Low LL, Yang Y, Lee KH. Applicability of a previously validated readmission predictive index in medical patients in Singapore: a retrospective study. BMC Health Serv Res 2013; 13 (01) 366
• 46 Singh D, Ham C. Improving care for people with long-term conditions: a review of UK and international frameworks. The King's Fund Information and Library Service. 2015 . Accessed May 2020 at: http://www.improvingchroniccare.org/downloads/review_of_international_frameworks__chris_hamm.pdf
• 47 Demir E. A decision support tool for predicting patients at risk of readmission: a comparison of classification trees, logistic regression, generalized additive models, and multivariate adaptive regression splines [internet]. University of Hertfordshire Business School. 2012 . Accessed May 2020 at: http://uhra.herts.ac.uk/bitstream/handle/2299/8985/s141.pdf?sequence=1
• 48 Ministry of Health, Singapore. Technical Manual for Indicators in the Public Hospital Performance Report. 2018
• 49 Moh.gov.sg. News highlights: Reorganisation of Healthcare System into three Integrated Clusters to better meet future healthcare needs. 2017 . Accessed May 2020 at: https://www.moh.gov.sg/news-highlights/details/reorganisation-of-healthcare-system-into-three-integrated-clusters-to-better-meet-future-healthcare-needs
• 50 Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol 1992; 45 (06) 613-619
• 51 Quan H, Sundararajan V, Halfon P. et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care 2005; 43 (11) 1130-1139
• 52 Low LL, Wah W, Ng MJ, Tan SY, Liu N, Lee KH. Housing as a social determinant of health in singapore and its association with readmission risk and increased utilization of hospital services. Front Public Health 2016; 4: 109
• 53 Hdb.gov.sg. Public rental scheme. Accessed May 2020 at: https://www.hdb.gov.sg/cs/infoweb/residential/renting-a-flat/renting-from-hdb/public-rental-scheme
• 54 Moh.gov.sg. Medifund, Ministry of Health [internet]. 2015 . Accessed May 2020 at: https://www.moh.gov.sg/content/moh_web/home/costs_and_financing/schemes_subsidies/Medifund.html
• 55 Moh.gov.sg. Hospital Services, Ministry of Health [Internet]. [cited 20 December 2019]. Accessed May 2020 at: https://www.moh.gov.sg/home/our-healthcare-system/healthcare-services-and-facilities/hospital-services
• 56 Ng Teng Fong General Hospital. Medical Social Services. [cited 20 December 2019]. Accessed May 2020 at: https://www.ntfgh.com.sg/AlliedHealthDetail/Medical_Social_Services
• 57 Sperandei S. Understanding logistic regression analysis. Biochem Med (Zagreb) 2014; 24 (01) 12-18
• 58 Natekin A, Knoll A. Gradient boosting machines, a tutorial. Front Neurorobot 2013; 7: 21
• 59 Ciampi A. Generalized regression trees. Comput Stat Data Anal 1991; 12: 57-78
• 60 Kingsford C, Salzberg SL. What are decision trees?. Nat Biotechnol 2008; 26 (09) 1011-1013
• 61 Song YY, Lu Y. Decision tree methods: applications for classification and prediction. Shanghai Jingshen Yixue 2015; 27 (02) 130-135
• 62 Breiman L. Random forests. Mach Learn 2001; 45 (01) 5-32
• 63 Antoniadi AM, Galvin M, Heverin M, Hardiman O, Mooney C. Prediction of caregiver burden in amyotrophic lateral sclerosis: a machine learning approach using random forests applied to a cohort study. BMJ Open 2020; 10 (02) e033109
• 64 Kumar R, Indrayan A. Receiver operating characteristic (ROC) curve for medical researchers. Indian Pediatr 2011; 48 (04) 277-287
• 65 Borra S, Di Ciaccio A. Measuring the prediction error. A comparison of cross-validation, bootstrap and covariance penalty methods. Comput Stat Data Anal 2010; 54: 2976-2989
• 66 Baillie CA, VanZandbergen C, Tait G. et al. The readmission risk flag: using the electronic health record to automatically identify patients at risk for 30-day readmission. J Hosp Med 2013; 8 (12) 689-695
• 67 Kripalani S, LeFevre F, Phillips CO, Williams MV, Basaviah P, Baker DW. Deficits in communication and information transfer between hospital-based and primary care physicians: implications for patient safety and continuity of care. JAMA 2007; 297 (08) 831-841
• 68 Gleason KM, McDaniel MR, Feinglass J. et al. Results of the Medications at Transitions and Clinical Handoffs (MATCH) study: an analysis of medication reconciliation errors and risk factors at hospital admission. J Gen Intern Med 2010; 25 (05) 441-447
• 69 Dudas V, Bookwalter T, Kerr KM, Pantilat SZ. The impact of follow-up telephone calls to patients after hospitalization. Am J Med 2001; 111 (9B) 26S-30S
• 70 Hernandez AF, Greiner MA, Fonarow GC. et al. Relationship between early physician follow-up and 30-day readmission among Medicare beneficiaries hospitalized for heart failure. JAMA 2010; 303 (17) 1716-1722
• 71 Nuhs.edu.sg. National University Health System. Accessed May 2020 at: https://www.nuhs.edu.sg/Care-in-the-Community/Getting-Well/Pages/default.aspx
• 72 Howard M, Bernard C, Tan A, Slaven M, Klein D, Heyland DK. Advance care planning: let's start sooner. Can Fam Physician 2015; 61 (08) 663-665
• 73 Kaboli PJ, Go JT, Hockenberry J. et al. Associations between reduced hospital length of stay and 30-day readmission rate and mortality: 14-year experience in 129 Veterans Affairs hospitals. Ann Intern Med 2012; 157 (12) 837-845
• 74 Kuhle S, Maguire B, Zhang H. et al. Comparison of logistic regression with machine learning methods for the prediction of fetal growth abnormalities: a retrospective cohort study. BMC Pregnancy Childbirth 2018; 18 (01) 333
• 75 Van Belle VM, Van Calster B, Timmerman D. et al. A mathematical model for interpretable clinical decision support with applications in gynecology. PLoS One 2012; 7 (03) e34312
• 76 Ehwerhemuepha L, Rakovski C. A comprehensive assessment of automatic logistic regression model selection methods. Accessed 2019 at: https://www.researchsquare.com/article/rs-7597/v1
• 77 Heinze G, Wallisch C, Dunkler D. Variable selection: a review and recommendations for the practicing statistician. Biom J 2018; 60 (03) 431-449
• 78 Rothman MJ, Rothman SI, Beals IV J. Development and validation of a continuous measure of patient condition using the Electronic Medical Record. J Biomed Inform 2013; 46 (05) 837-848
• 79 Cai X, Perez-Concha O, Coiera E. et al. Real-time prediction of mortality, readmission, and length of stay using electronic health record data. J Am Med Inform Assoc 2016; 23 (03) 553-561
• 80 Hasan O, Meltzer DO, Shaykevich SA. et al. Hospital readmission in general medicine patients: a prediction model. J Gen Intern Med 2010; 25 (03) 211-219
• 81 Cui Y, Metge C, Ye X, Moffatt M, Oppenheimer L, Forget EL. Development and validation of a predictive model for all-cause hospital readmissions in Winnipeg, Canada. J Health Serv Res Policy 2015; 20 (02) 83-91
• 82 López-Aguilà S, Contel JC, Farré J, Campuzano JL, Rajmil L. Predictive model for emergency hospital admission and 6-month readmission. Am J Manag Care 2011; 17 (09) e348-e357
• 83 Escobar GJ, Ragins A, Scheirer P, Liu V, Robles J, Kipnis P. Nonelective rehospitalizations and postdischarge mortality: predictive models suitable for use in real time. Med Care 2015; 53 (11) 916-923
• 84 Fu L, Li FM, Zhou J. Event prediction in healthcare analytics: beyond prediction accuracy. In: Cao HP, Li JY, Wang RL. eds. Trends and Applications in Knowledge Discovery and Data Mining: PAKDD 2016 Workshops, BDM, MLSDA, PACC, WDMBF, Auckland, New Zealand, April 19, 2016, Revised Selected Papers. Cham, Switzerland: Springer; 2016
• 85 Gottesman O, Kuivaniemi H, Tromp G. et al; eMERGE Network. The Electronic Medical Records and Genomics (eMERGE) Network: past, present, and future. Genet Med 2013; 15 (10) 761-771