Is Propensity Score Analysis a Valid Surrogate of Randomization for the Avoidance of Allocation Bias?

Abstract

Randomized clinical trials are the gold standard when experimental designs are feasible. Randomization allows the handling of allocation bias for known and unknown confounders. Specific tools such as blocking, stratification, and dynamic allocation provide additional guarantees to simple randomization. When an experimental design is not feasible, the propensity score (PS) has been shown to produce greater benefit than traditional methods (i.e., restriction, stratification, matching and adjusting). There appears to be a hierarchy in terms of the effectiveness of balancing for PS techniques: matching or weighting above stratification above covariate adjustment (which is discouraged due to its drawbacks). Instrumental variable analysis and its variants might provide added value because they aim to balance for unknown confounders as well, thus mimicking randomization, but at present, are considered more useful for sensitivity rather than primary analyses.

Keywords

allocation bias - confounding - randomization - propensity scores - instrumental variable analysis

• References

• 1 Goodman SN, Schneeweiss S, Baiocchi M. Using design thinking to differentiate useful from misleading evidence in observational research. JAMA 2017; 317 (07) 705-707
• 2 Pocock SJ, Elbourne DR. Randomized trials or observational tribulations?. N Engl J Med 2000; 342 (25) 1907-1909
• 3 Sedgwick P. Selection bias versus allocation bias. BMJ 2013; 346: f3345
• 4 Feinstein AR. Epidemiologic analyses of causation: the unlearned scientific lessons of randomized trials. J Clin Epidemiol 1989; 42 (06) 481-489 , discussion 499–502
• 5 Moses LE. Measuring effects without randomized trials? Options, problems, challenges. Med Care 1995; 33 (4, Suppl) AS8-AS14
• 6 Kunz R, Oxman AD. The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials. BMJ 1998; 317 (7167): 1185-1190
• 7 Byar DP. Why data bases should not replace randomized clinical trials. Biometrics 1980; 36 (02) 337-342
• 8 Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med 2000; 342 (25) 1887-1892
• 9 Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med 2000; 342 (25) 1878-1886
• 10 Giordano SH, Kuo Y-F, Duan Z, Hortobagyi GN, Freeman J, Goodwin JS. Limits of observational data in determining outcomes from cancer therapy. Cancer 2008; 112 (11) 2456-2466
• 11 McGale P, Cutter D, Darby SC, Henson KE, Jagsi R, Taylor CW. Can observational data replace randomized trials?. J Clin Oncol 2016; 34 (27) 3355-3357
• 12 Chalmers TC, Celano P, Sacks HS, Smith Jr H. Bias in treatment assignment in controlled clinical trials. N Engl J Med 1983; 309 (22) 1358-1361
• 13 Naylor CD, Guyatt GH. ; The Evidence-Based Medicine Working Group. Users' guides to the medical literature. X. How to use an article reporting variations in the outcomes of health services. JAMA 1996; 275 (07) 554-558
• 14 Hernán MA, Alonso A, Logan R. , et al. Observational studies analyzed like randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease. Epidemiology 2008; 19 (06) 766-779
• 15 Sherman RE, Anderson SA, Dal Pan GJ. , et al. Real-world evidence - what is it and what can it tell us?. N Engl J Med 2016; 375 (23) 2293-2297
• 16 Johnson ML, Crown W, Martin BC, Dormuth CR, Siebert U. Good research practices for comparative effectiveness research: analytic methods to improve causal inference from nonrandomized studies of treatment effects using secondary data sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report--Part III. Value Health 2009; 12 (08) 1062-1073
• 17 Berger ML, Mamdani M, Atkins D, Johnson ML. Good research practices for comparative effectiveness research: defining, reporting and interpreting nonrandomized studies of treatment effects using secondary data sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report--Part I. Value Health 2009; 12 (08) 1044-1052
• 18 Cox E, Martin BC, Van Staa T, Garbe E, Siebert U, Johnson ML. Good research practices for comparative effectiveness research: approaches to mitigate bias and confounding in the design of nonrandomized studies of treatment effects using secondary data sources: the International Society for Pharmacoeconomics and Outcomes Research Good Research Practices for Retrospective Database Analysis Task Force Report--Part II. Value Health 2009; 12 (08) 1053-1061
• 19 Agoritsas T, Merglen A, Shah ND, O'Donnell M, Guyatt GH. Adjusted analyses in studies addressing therapy and harm: users' guides to the medical literature. JAMA 2017; 317 (07) 748-759
• 20 Jager KJ, Zoccali C, Macleod A, Dekker FW. Confounding: what it is and how to deal with it. Kidney Int 2008; 73 (03) 256-260
• 21 Sedgwick P. Confounding in clinical trials. BMJ 2012; 345: e7951
• 22 van Stralen KJ, Dekker FW, Zoccali C, Jager KJ. Confounding. Nephron Clin Pract 2010; 116 (02) c143-c147
• 23 Armitage P. The role of randomization in clinical trials. Stat Med 1982; 1 (04) 345-352
• 24 Peto R. Why do we need systematic overviews of randomized trials?. Stat Med 1987; 6 (03) 233-244
• 25 Altman DG, Bland JM. Statistics notes. Treatment allocation in controlled trials: why randomise?. BMJ 1999; 318 (7192): 1209
• 26 Roberts C, Torgerson DJ. Understanding controlled trials: baseline imbalance in randomised controlled trials. BMJ 1999; 319 (7203): 185
• 27 Stel VS, Dekker FW, Zoccali C, Jager KJ. Instrumental variable analysis. Nephrol Dial Transplant 2013; 28 (07) 1694-1699
• 28 Austin PC. A critical appraisal of propensity-score matching in the medical literature between 1996 and 2003. Stat Med 2008; 27 (12) 2037-2049
• 29 Austin PC. Propensity-score matching in the cardiovascular surgery literature from 2004 to 2006: a systematic review and suggestions for improvement. J Thorac Cardiovasc Surg 2007; 134 (05) 1128-1135
• 30 Austin PC. Primer on statistical interpretation or methods report card on propensity-score matching in the cardiology literature from 2004 to 2006: a systematic review. Circ Cardiovasc Qual Outcomes 2008; 1 (01) 62-67
• 31 Schulz KF, Grimes DA. Generation of allocation sequences in randomised trials: chance, not choice. Lancet 2002; 359 (9305): 515-519
• 32 Fisher RA. The arrangement of field experiments. J Minist Agric 1926; 33: 503-513
• 33 Hill A. Principles of medical statistics. Lancet 1937; 229 (5925): 706-708
• 34 Doll R. Controlled trials: the 1948 watershed. BMJ 1998; 317 (7167): 1217-1220
• 35 Streptomycin in Tuberculosis Trials Committee. Streptomycin treatment of pulmonary tuberculosis. BMJ 1948; 2 (4582): 769-782
• 36 Medical Research Council (UK). Prevention of whooping-cough by vaccination; a Medical Research Council investigation. BMJ 1951; 1 (4721): 1463-1471
• 37 Lachin JM, Matts JP, Wei LJ. Randomization in clinical trials: conclusions and recommendations. Control Clin Trials 1988; 9 (04) 365-374
• 38 Egbewale BE. Random allocation in controlled clinical trials: a review. J Pharm Pharm Sci 2014; 17 (02) 248-253
• 39 Hewitt CE, Torgerson DJ. Is restricted randomisation necessary?. BMJ 2006; 332 (7556): 1506-1508
• 40 Doig GS, Simpson F. Randomization and allocation concealment: a practical guide for researchers. J Crit Care 2005; 20 (02) 187-191 , discussion 191–193
• 41 Sedgwick P. Treatment allocation in trials: stratified randomisation. BMJ 2015; 350: h978
• 42 Miettinen OS. Stratification by a multivariate confounder score. Am J Epidemiol 1976; 104 (06) 609-620
• 43 Grizzle JE. A note on stratifying versus complete random assignment in clinical trials. Control Clin Trials 1982; 3 (04) 365-368
• 44 Kernan WN, Viscoli CM, Makuch RW, Brass LM, Horwitz RI. Stratified randomization for clinical trials. J Clin Epidemiol 1999; 52 (01) 19-26
• 45 Tu D, Shalay K, Pater J. Adjustment of treatment effect for covariates in clinical trials: statistical and regulatory issues. Drug Inf J 2000; 34 (02) 511-523
• 46 Roberts C, Torgerson D. Randomisation methods in controlled trials. BMJ 1998; 317 (7168): 1301
• 47 Bracken MB. On stratification, minimization and protection against types 1 and 2 error. J Clin Epidemiol 2001; 54 (01) 104-105
• 48 Rosenberger W, Lachin J. Randomization in clinical trials: theory and practice. New York, NY: Wiley; 2002
• 49 Hallstrom A, Davis K. Imbalance in treatment assignments in stratified blocked randomization. Control Clin Trials 1988; 9 (04) 375-382
• 50 Taves DR. Minimization: a new method of assigning patients to treatment and control groups. Clin Pharmacol Ther 1974; 15 (05) 443-453
• 51 Pocock S. Clinical Trials: A Practical Approach. 2nd ed. New York: Wiley; 1983
• 52 White SJ, Freedman LS. Allocation of patients to treatment groups in a controlled clinical study. Br J Cancer 1978; 37 (05) 849-857
• 53 Kahan BC, Rehal S, Cro S. Risk of selection bias in randomised trials. Trials 2015; 16 (01) 405
• 54 Pocock SJ, Simon R. Sequential treatment assignment with balancing for prognostic factors in the controlled clinical trial. Biometrics 1975; 31 (01) 103-115
• 55 Cai H, Xia J, Xu D, Gao D, Yan Y. A generic minimization random allocation and blinding system on web. J Biomed Inform 2006; 39 (06) 706-719
• 56 International Conference on Harmonization (ICH E9). Statistical Principles for Clinical Trials: 1998. Available at http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E9/Step4/E9_Guideline.pdf . Accessed August 6, 2017
• 57 Scott NW, McPherson GC, Ramsay CR, Campbell MK. The method of minimization for allocation to clinical trials. A review. Control Clin Trials 2002; 23 (06) 662-674
• 58 Hagino A, Hamada C, Yoshimura I, Ohashi Y, Sakamoto J, Nakazato H. Statistical comparison of random allocation methods in cancer clinical trials. Control Clin Trials 2004; 25 (06) 572-584
• 59 Lin Y, Zhu M, Su Z. The pursuit of balance: An overview of covariate-adaptive randomization techniques in clinical trials. Contemp Clin Trials 2015; 45 (Pt A): 21-25
• 60 Birkett NJ. Adaptive allocation in randomized controlled trials. Control Clin Trials 1985; 6 (02) 146-155
• 61 Therneau TM. How many stratification factors are “too many” to use in a randomization plan?. Control Clin Trials 1993; 14 (02) 98-108
• 62 Signorini DF, Leung O, Simes RJ, Beller E, Gebski VJ, Callaghan T. Dynamic balanced randomization for clinical trials. Stat Med 1993; 12 (24) 2343-2350
• 63 Heritier S, Gebski V, Pillai A. Dynamic balancing randomization in controlled clinical trials. Stat Med 2005; 24 (24) 3729-3741
• 64 Halpern J, Brown Jr BW. Sequential treatment allocation procedures in clinical trials--with particular attention to the analysis of results for the biased coin design. Stat Med 1986; 5 (03) 211-229
• 65 Kalish LA, Begg CB. Treatment allocation methods in clinical trials: a review. Stat Med 1985; 4 (02) 129-144
• 66 Simon R. Restricted randomization designs in clinical trials. Biometrics 1979; 35 (02) 503-512
• 67 Green H, McEntegart DJ, Byrom B, Ghani S, Shepherd S. Minimization in crossover trials with non-prognostic strata: theory and practical application. J Clin Pharm Ther 2001; 26 (02) 121-128
• 68 Shao J, Yu X, Zhong B. A theory for testing hypotheses under covariate-adaptive randomization. Biometrika 2010; 97 (02) 347-360
• 69 Shao J, Yu X. Validity of tests under covariate-adaptive biased coin randomization and generalized linear models. Biometrics 2013; 69 (04) 960-969
• 70 Treasure T, MacRae KD. Minimisation: the platinum standard for trials? Randomisation doesn't guarantee similarity of groups; minimisation does. BMJ 1998; 317 (7155): 362-363
• 71 Moher D, Hopewell S, Schulz KF. , et al; Consolidated Standards of Reporting Trials Group. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. J Clin Epidemiol 2010; 63 (08) e1-e37
• 72 Taves DR. The use of minimization in clinical trials. Contemp Clin Trials 2010; 31 (02) 180-184
• 73 Buyse M, McEntegart DJ. Achieving balance in clinical trials: an unbalanced view from EU regulators. Appl Clin Trials 2004; 13: 36-40
• 74 Rosenberger WF, Sverdlov O. Handling covariates in the design of clinical trials. Stat Sci 2008; 23 (03) 404-419
• 75 Barbáchano Y, Coad DS. Inference following designs which adjust for imbalances in prognostic factors. Clin Trials 2013; 10 (04) 540-551
• 76 Aickin M. Effect of design-adaptive allocation on inference for a regression parameter: two-group, single-covariate and double-covariate cases. Stat Probabil Lett 2009; 79 (12) 16-20
• 77 Assmann SF, Pocock SJ, Enos LE, Kasten LE. Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet 2000; 355 (9209): 1064-1069
• 78 Schulz KF, Altman DG, Moher D. ; CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. Trials 2010; 11 (01) 32
• 79 Altman DG, Doré CJ. Randomisation and baseline comparisons in clinical trials. Lancet 1990; 335 (8682): 149-153
• 80 Altman D. Adjustment for covariate imbalance. In: Armitage PCT. , ed. Encyclopedia of Biostatistics. Chichester, UK: Wiley; 1998: 1000-1005
• 81 Altman DG. Comparability of randomised groups. J R Stat Soc Ser D Stat 1985; 34 (01) 125-136
• 82 Schulz KF, Chalmers I, Grimes DA, Altman DG. Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals. JAMA 1994; 272 (02) 125-128
• 83 Senn SJ. Base logic: tests of baseline balance in randomized clinical trials. Clin Res & Reg Affairs 1995; 12 (03) 171-182
• 84 Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res 2011; 46 (03) 399-424
• 85 Austin PC. A tutorial and case study in propensity score analysis: an application to estimating the effect of in-hospital smoking cessation counseling on mortality. Multivariate Behav Res 2011; 46 (01) 119-151
• 86 Deb S, Austin PC, Tu JV. , et al. A review of propensity-score methods and their use in cardiovascular research. Can J Cardiol 2016; 32 (02) 259-265
• 87 D'Agostino Jr RB. Propensity scores in cardiovascular research. Circulation 2007; 115 (17) 2340-2343
• 88 D'Agostino Jr RB. Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Stat Med 1998; 17 (19) 2265-2281
• 89 Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika 1983; 70 (01) 41-55
• 90 Baser O. Too much ado about propensity score models? Comparing methods of propensity score matching. Value Health 2006; 9 (06) 377-385
• 91 Haukoos JS, Lewis RJ. The Propensity Score. JAMA 2015; 314 (15) 1637-1638
• 92 Rubin DB, Thomas N. Matching using estimated propensity scores: relating theory to practice. Biometrics 1996; 52 (01) 249-264
• 93 Heinze G, Jüni P. An overview of the objectives of and the approaches to propensity score analyses. Eur Heart J 2011; 32 (14) 1704-1708
• 94 Imai K, King G, Stuart EA. Misunderstandings between experimentalists and observationalists about causal inference. J R Stat Soc A. 2008; 171 (02) 481-502
• 95 Austin PC. Goodness-of-fit diagnostics for the propensity score model when estimating treatment effects using covariate adjustment with the propensity score. Pharmacoepidemiol Drug Saf 2008; 17 (12) 1202-1217
• 96 Austin PC. Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Stat Med 2009; 28 (25) 3083-3107
• 97 Nguyen T-L, Collins GS, Spence J. , et al. Double-adjustment in propensity score matching analysis: choosing a threshold for considering residual imbalance. BMC Med Res Methodol 2017; 17 (01) 78
• 98 Austin PC, Grootendorst P, Anderson GM. A comparison of the ability of different propensity score models to balance measured variables between treated and untreated subjects: a Monte Carlo study. Stat Med 2007; 26 (04) 734-753
• 99 Westreich D, Cole SR, Funk MJ, Brookhart MA, Stürmer T. The role of the c-statistic in variable selection for propensity score models. Pharmacoepidemiol Drug Saf 2011; 20 (03) 317-320
• 100 Rosenbaum PR. Model-based direct adjustment. J Am Stat Assoc 1987; 82 (398) 387
• 101 Robins JM, Hernán MA, Brumback B. Marginal structural models and causal inference in epidemiology. Epidemiology 2000; 11 (05) 550-560
• 102 D'Agostino Jr RB. Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Stat Med 1998; 17 (19) 2265-2281
• 103 Imbens GW. Nonparametric estimation of average treatment effects under exogeneity: a review. Rev Econ Stat 2004; 86 (01) 4-29
• 104 Austin PC, Grootendorst P, Normand S-LT, Anderson GM. Conditioning on the propensity score can result in biased estimation of common measures of treatment effect: a Monte Carlo study. Stat Med 2007; 26 (04) 754-768
• 105 Austin PC. Type I error rates, coverage of confidence intervals, and variance estimation in propensity-score matched analyses. Int J Biostat 2009; 5 (01) 13
• 106 Austin PC, Mamdani MM. A comparison of propensity score methods: a case-study estimating the effectiveness of post-AMI statin use. Stat Med 2006; 25 (12) 2084-2106
• 107 Lunceford JK, Davidian M. Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study. Stat Med 2004; 23 (19) 2937-2960
• 108 Austin PC. The performance of different propensity score methods for estimating marginal hazard ratios. Stat Med 2013; 32 (16) 2837-2849
• 109 Austin PC. The performance of different propensity score methods for estimating marginal odds ratios. Stat Med 2007; 26 (16) 3078-3094
• 110 Austin PC. The relative ability of different propensity score methods to balance measured covariates between treated and untreated subjects in observational studies. Med Decis Making 2009; 29 (06) 661-677
• 111 Garrido MM, Kelley AS, Paris J. , et al. Methods for constructing and assessing propensity scores. Health Serv Res 2014; 49 (05) 1701-1720
• 112 Bergstralh EJ, Kosanke JL, Jacobsen SJ. Software for optimal matching in observational studies. Epidemiology 1996; 7 (03) 331-332
• 113 Rubin DB. The use of matched sampling and regression adjustment to remove bias in observational studies. Biometrics 1973; 29 (01) 185
• 114 Rubin DB. Using multivariate matched sampling and regression adjustment to control bias in observational. ETS Res Bull Ser 1978; 02: i-33
• 115 Alemayehu D, Alvir JMJ, Jones B, Willke RJ. Statistical issues with the analysis of nonrandomized studies in comparative effectiveness research. J Manag Care Pharm 2011; 17 (9, Suppl A) S22-S26
• 116 Heckman J. The common structure of statistical models of truncation, sample selection, and limited dependent variables and an estimator for such models. In: Annals of Economic and Social Measurement. Vol 5. National Bureau of Economic Research, Inc 1976; 5 (04) 475-492
• 117 Bagiella E, Karamlou T, Chang H, Spivack J. Instrumental variable methods in clinical research. J Thorac Cardiovasc Surg 2015; 150 (04) 779-782
• 118 Klungel OH, Jamal Uddin M, de Boer A. , et al. Instrumental variable analysis in epidemiologic studies: an overview of the estimation methods. Pharm Anal Acta 2015; 6: 353
• 119 Groenwold RHH, Uddin MJ, Roes KCB. , et al. Instrumental variable analysis in randomized trials with non-compliance and observational pharmacoepidemiologic studies. OA Epidemiol 2014; 2: 1-9
• 120 Ertefaie A, Small DS, Flory JH, Hennessy S. A tutorial on the use of instrumental variables in pharmacoepidemiology. Pharmacoepidemiol Drug Saf 2017; 26 (04) 357-367
• 121 Baiocchi M, Cheng J, Small DS. Instrumental variable methods for causal inference. Stat Med 2014; 33 (13) 2297-2340
• 122 Stürmer T, Joshi M, Glynn RJ, Avorn J, Rothman KJ, Schneeweiss S. A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. J Clin Epidemiol 2006; 59 (05) 437-447
• 123 Weitzen S, Lapane KL, Toledano AY, Hume AL, Mor V. Principles for modeling propensity scores in medical research: a systematic literature review. Pharmacoepidemiol Drug Saf 2004; 13 (12) 841-853
• 124 Shah BR, Laupacis A, Hux JE, Austin PC. Propensity score methods gave similar results to traditional regression modeling in observational studies: a systematic review. J Clin Epidemiol 2005; 58 (06) 550-559
• 125 Davies NM, Smith GD, Windmeijer F, Martin RM. Issues in the reporting and conduct of instrumental variable studies: a systematic review. Epidemiology 2013; 24 (03) 363-369
• 126 Chen Y, Briesacher BA. Use of instrumental variable in prescription drug research with observational data: a systematic review. J Clin Epidemiol 2011; 64 (06) 687-700
• 127 Ali MS, Groenwold RHH, Belitser SV. , et al. Reporting of covariate selection and balance assessment in propensity score analysis is suboptimal: a systematic review. J Clin Epidemiol 2015; 68 (02) 112-121
• 128 Laborde-Castérot H, Agrinier N, Thilly N. Performing both propensity score and instrumental variable analyses in observational studies often leads to discrepant results: a systematic review. J Clin Epidemiol 2015; 68 (10) 1232-1240
• 129 Smith GD, Ebrahim S. ‘Mendelian randomization’: can genetic epidemiology contribute to understanding environmental determinants of disease?. Int J Epidemiol 2003; 32 (01) 1-22
• 130 Boef AGC, Dekkers OM, le Cessie S. Mendelian randomization studies: a review of the approaches used and the quality of reporting. Int J Epidemiol 2015; 44 (02) 496-511
• 131 Burgess S, Small DS, Thompson SG. A review of instrumental variable estimators for Mendelian randomization. Stat Methods Med Res 2015; pii: 0962280215597579 . [Epub ahead of print]
• 132 Smith GD. Mendelian randomization for strengthening causal inference in observational studies: application to gene × environment interactions. Perspect Psychol Sci 2010; 5 (05) 527-545