Statistical Process Control for Monitoring Standardized Mortality Ratios of a Classification Tree Model

Summary

Objectives: The ratio of observed to expected mortality (standardized mortality ratio, SMR), is a key indicator of quality of care. We use PreControl Charts to investigate SMR behavior over time of an existing tree-model for predicting mortality in intensive care units (ICUs) and its implications for hospital ranking. We compare the results to those of a logistic regression model.

Methods: We calculated SMRs of 30 equally-sized consecutive subsets from a total of 12,143 ICU patients aged 80 years or older and plotted them on a PreControl Chart. We calculated individual hospital SMRs in 2009, with and without repeated recalibration of the models on earlier data.

Results: The overall SMR of the tree-model was stable over time, in contrast to logistic regression. Both models were stable after repeated recalibration. The overall SMR of the tree on the whole validation set was statistically significantly different (SMR 1.00 ± 0.012 vs. 0.94 ± 0.01) and worse in performance than the logistic regression model (AUC 0.76 ± 0.005 vs. 0.79 ± 0.004; Brier score 0.17 ± 0.012 vs. 0.16 ± 0.010). The individual SMRs’ range in 2009 was 0.53–1.31 for the tree and 0.64–1.27 for logistic regression. The proportion of individual hospitals with SMR >1, hinting at poor quality of care, reduced from 38% to 29% after recalibration for the tree, and increased from 15% to 35% for logistic regression.

Conclusions: Although the tree-model has seemingly a longer shelf life than the logistic regression model, its SMR may be less useful for quality of care assessment as it insufficiently responds to changes in the population over time.

Keywords

Prognostic models - intensive care - predictive performance - temporal validation - statistical process control

• References

• 1 Zimmerman D. Benchmarking: measuring yourself against the best. Trustee 1999; 52: 22-13.
• 2 Zimmerman JE, Alzola C, Von Rueden KT. The use of benchmarking to identify top performing critical care units: a preliminary assessment of their policies and practices. J Crit Care 2003; 18: 76-86.
• 3 Kutschmann M, Renner D, Fischer B. Use of Logistic Regression for Developing Risk-adjusted Quality Indicators in Nursing. Methods Inf Med 2010; 49 (06) 608-612.
• 4 Fleis J, Levin B, Paik M. Statistical methods for rates and proportions. New Jersey: J Wiley; 2003.
• 5 de Jonge E, Bosman RJ, van der Voort PH, Korsten HH, Scheffer GJ, de Keizer NF. Intensive care medicine in the Netherlands, 1997-2001. I. Patient population and treatment outcome. Ned Tijdschr Geneeskd 2003; 147: 1013-1017.
• 6 Bellazzi R, Diomidous M, Sarkar IN, Takabayashi K, Ziegler A, McCray AT. Data analysis and data mining: current issues in biomedical informatics. Methods Inf Med 2011; 50 (06) 536-544.
• 7 Altman DG, Vergouwe Y, Royston P, Moons KG. Prognosis and prognostic research: validating a prognostic model. BMJ 2009; 338: b605
• 8 Minne L, Eslami S, de Keizer NF, de Jonge E, de Rooij SE, Abu-Hanna A. Effect of changes over time in the performance of a customized SAPS-II model on quality of care assessment. Intensive Care Med 2012; 38 (01) 40-46.
• 9 R Development Core Team R: A Language and Environment for Statistical Computing Vienna 2009.
• 10 de Rooij SE, Abu-Hanna A, Levi M, de Jonge E. Identification of high-risk subgroups in very elderly intensive care unit patients. Crit Care 2007; 11: R33
• 11 Breiman L, Friedman JH, Olshen RA, Stone CJ. Classification and regression trees. Wadsworth: CRC Press; 1984.
• 12 Altman DG. Practical statistics for medical research. London: Chapman and Hall; 1990.
• 13 Hand DJ. Construction and Assessment of Classification Rules. New York: John Wiley & Sons; 1997.
• 14 Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 1993; 270: 2957-2963.
• 15 Cher DJ, Deubner DC, Kelsh MA, Chapman PS, Ray RM. Assessment of the beryllium lymphocyte proliferation test using statistical process control. Inhal Toxicol 2006; 18: 901-910.
• 16 Carey RG. How do you know that your chart is improving? Part I: Basic concepts in statistical thinking. J Ambulatory Care Manage 2002; 25: 80-87.
• 17 Wheeler JW. Chapter Sixteen Miscellaneous Techniques. Advanced Topics in Statistical Process Control. Knoxville, Tennessee: Spc Pr.; 2004.
• 18 Jarman B, Bottle A, Aylin P. Monitoring changes in hospital standardised mortality ratios. BMJ 2005; 7487: 329-330.
• 19 Wright J, Dugdale B, Hammons I, Jarman B, Neary M, Newton D. et al. Learning from death: a hospital mortality reduction programme. J R Soc Med 2006; 99 (06) 303-308.
• 20 Le Gall JR, Neumann A, Hemery F, Bleriot JP, Fulgencio JP, Garrigues B. et al. Mortality prediction using SAPS II: an update for French intensive care units. Crit Care 2005; 9: R645-R652.
• 21 Steyerberg EW. Clinical Prediction Models. A practical approach to development, validation, and updating. New York: Springer; 2009.
• 22 Steyerberg EW, Borsboom GJ, van Houwelingen HC, Eijkemans MJ, Habbema JD. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med 2004; 23: 2567-2586.
• 23 Harrison DA, Brady AR, Parry GJ, Carpenter JR, Rowan K. Recalibration of risk prediction models in a large multicenter cohort of admissions to adult, general critical care units in the United Kingdom. Crit Care Med 2006; 34: 1378-1388.
• 24 Bakhshi-Raiez F, Peek N, Bosman RJ, de Jonge E, de Keizer NF. The impact of different prognostic models and their customization on institutional comparison of intensive care units. Crit Care Med 2007; 35: 2553-2560.