Multiplicity Issues in Microarray Experiments

Summary

Objectives: Discussion of different error concepts relevant to microarray experiments. Review of some commonly used multiple testing procedures. Comparison of different approaches as applied to gene expression data.

Methods: This article focuses on familywise error rate (FWER) and false discovery rate (FDR) controlling procedures. Methods under investigation include: Bonferroni-type methods and their improvements (including resampling approaches), modified Bonferroni methods, data-driven approaches, as well as the linear step-up method and its modifications. Particular emphasis lies on the description of the assumptions, advantages and limitations for the investigated methods.

Results: FWER controlling procedures are often too conservative in high dimensional screening studies. A better balance between the raw P-values and the stringent FWER-adjusted P-values may be required in many situations, as provided by FDR controlling and related procedures.

Conclusions: The questions remain open, which error concept to apply and which multiple testing procedure to use. Although we believe that the FDR or one of its variants will be applied more often in the future, longterm experience with microarray technology is missing and thus the validity of appropriate multiple test procedures cannot yet be assessed for microarray data analysis.

Keywords

Familywise error rate - false discovery rate - error concepts - screening

• References

• 1 Hochberg Y, Tamhane AC. Multiple comparison procedures. New York: Wiley; 1987
• 2 Holm S. A simple sequentially multiple test procedure. Scand J Stat 1979; 6: 65-70.
• 3 Shaffer JP. Modified sequentially rejective multiple test procedures. J Am Stat Assoc 1986; 81: 826-31.
• 4 Westfall PH, Young SS. Resampling-based multiple testing. New York: Wiley; 1993
• 5 Dudoit S, Shaffer JP, Boldrick JC. Multiple hypothesis testing in microarray experiments. Stat Sci 2003; 18: 71-103.
• 6 Sarkar SK, Chang CK. Simes’ method for multiple hypothesis testing with positively dependent test statistics. J Am Stat Assoc 1997; 92: 1601-8.
• 7 Simes RJ. An improved Bonferroni procedure for multiple tests of significance. Biometrika 1986; 73: 751-4.
• 8 Hochberg Y. A sharper Bonferroni procedure for multiple tests of significance. Biometrika 1988; 75: 800-2.
• 9 Hommel G. A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 1988; 75: 383-6.
• 10 Westfall PH, Krishen A. Optimally weighted, fixed sequence and gatekeeping multiple testing procedures. J Stat Plan Infer 2001; 99: 25-40.
• 11 Kropf S, Läuter J. Multiple tests for different sets of variables using a data-driven ordering of hypotheses, with an application to gene expression data. Biom J 2002; 44: 789-800.
• 12 Läuter J, Glimm E, Kropf S. Multivariate tests based on left-spherically distributed linear scores. Ann Stat 1998; 26: 1972-88.
• 13 Westfall PH, Kropf S, Finos L. Weighted FWE controlling methods in high-dimensional situations. In Recent developments in multiple comparisons procedures. Benjamini Y, Bretz F, Sarkar S. (eds.) IMS Lecture Notes – Monograph Series, Vol 47. Beachwood, Ohio, USA: 2004: 143-54.
• 14 Hommel G, Kropf S. Tests for Differentiation in Gene Expression Using a Data-Driven Order or Weights for Hypotheses. Biometrical Journal 2005 (to appear).
• 15 Benjamini Y, Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J Roy Stat Soc B 1995; 57: 289-300.
• 16 Seeger P. A note on a method for the analysis of significance en masse. Technometrics 1968; 10: 586-93.
• 17 Finner H, Roters M. On the false discovery rate and expected type I errors. Biom J 2001; 43: 985-1005.
• 18 Troendle JF. Stepwise normal theory multiple test procedures controlling the false discovery rate. J Stat Plan Infer 2000; 84: 139-58.
• 19 Hsu JC, Chang JY, Wang T. Multiple comparisons in screening for differential gene expressions from microarray data. In. Screening Dean A, Lewis S. (eds.) New York: Springer-Verlag; 2004
• 20 Weller JI, Song JZ, Heyen DW, Lewin HA, Ron M. A new approach to the problem of multiple comparisons in the genetic dissection of complex traits. Genetics 1998; 150: 1699-1706.
• 21 Zaykin DV, Young SS, Westfall PH. Using false discovery rate approach in the genetic dissection of complex traits: a response to Weller, et al. Genetics 2000; 154: 1917-8.
• 22 Storey JD, Taylor JE, Siegmund D. Strong control, conservative point estimation, and simultaneous conservative consistency of false discovery rates: A unified approach. J Roy Stat Soc B 2004; 66: 187-205.
• 23 Storey JD. A direct approach to false discovery rates. J Roy Stat Soc B 2002; 64: 479-98.
• 24 Efron B, Tibshirani R, Storey JD, Tusher V. Empirical Bayes analysis of a microarray experiment. J Am Stat Assoc 2001; 96: 1151-60.
• 25 Korn EL, Troendle JF, McShane LM, Simon R. Controlling the number of false discoveries: Application to high dimensional genomic data. J Stat Plan Infe. 2004. (in press)
• 26 van der Laan MJ, Dudoit S, Pollard KS. Augmentation Procedures for Control of the Generalized Family-Wise Error Rate and Tail Probabilities for the Proportion of False Positives. Statistical Applications in Genetics and Molecular Biology. 2004: 3-15. June
• 27 Hsueh HM, Chen JJ, Kodell RL. Comparison of methods for estimating the number of true null hypotheses in multiplicity testing. J Biopharm Stat 2003; 13: 675-89.
• 28 Benjamini Y, Hochberg Y. The adaptive control of the false discovery rate in multiple hypotheses testing with independent statistics. J Educ Behav Stat 2000; 25: 60-83.
• 29 Benjamini Y, Yekutieli D. The control of the false discovery rate in multiple hypothesis testing under dependency. Ann Stat 2001; 29: 1165-88.
• 30 Yekutieli D, Benjamini Y. Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. J Stat Plan Infer 1999; 82: 171-96.
• 31 Somerville PN. FDR step-down and step-up procedures for the correlated case. In Recent developments in multiple comparisons procedures.. Benjamini Y, Bretz F, Sarkar S. (eds.) IMS Lecture Notes – Monograph Series, Vol. 47. Beachwood, Ohio, USA: 2004: 100-18.
• 32 Benjamini Y, Liu W. A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence. J Stat Plan Infer 1999; 82: 163-70.
• 33 Horn M, Dunnett CW. Power and sample size comparisons of stepwise FWE and FDR controlling test procedures in the normal many-one case. In Recent developments in multiple comparisons procedures. Benjamini Y, Bretz F, Sarkar S. (eds.) IMS Lecture Notes – Monograph Series, Vol. 47. Beachwood, Ohio, USA: 2004: 48-64.
• 34 Dudoit S, van der Laan MJ, Pollard KS. Multiple testing. Part I. Single-step procedures for control of general type I error rates. Statistical Applications in Genetics and Molecular Biology 2004; 3 June 14
• 35 van der Laan MJ, Dudoit S, Pollard KS. Multiple testing. Part II. Step-down procedures for control of the family-wise error rate. Statistical Applications in Genetics and Molecular Biology 2004; 3 June 9
• 36 Golub TR, Slonim DK, Tamayo P, Huard C, Gaasenbeek M, Mesirov JP, Coller H, Loh ML, Downing JR, Caligiuri MA, Bloomfield CD, Lander ES. Molecular classification of cancer: Class discovery and class prediction by gene expression monitoring. Science 1999; 286: 531-7.
• 37 Tusher VG, Tibshirani R, Chu G. Significance analysis of microarrays applied to ionizing radiation response. Proc Ntl Acad Sci 2001; 98: 5116-21.
• 38 Ge Y, Dudoit S, Speed T. Resampling-based multiple testing for microarray data analysis. Test 2003; 12: 1-77.