Download PDF
Methods Inf Med 2005; 44(03): 431-437
DOI: 10.1055/s-0038-1633989
Original Article
Schattauer GmbH

Multiplicity Issues in Microarray Experiments

F. Bretz
1   B&SR, Novartis Pharma AG, Basel, Switzerland
,
J. Landgrebe
2   Abteilung Biochemie II, Universität Göttingen, Göttingen, Germany
,
E. Brunner
3   Abteilung Medizinische Statistik, Universität Göttingen, Göttingen, Germany
› Author Affiliations
Further Information

Publication History

Publication Date:
06 February 2018 (online)

    Permissions and Reprints

Summary

Objectives: Discussion of different error concepts relevant to microarray experiments. Review of some commonly used multiple testing procedures. Comparison of different approaches as applied to gene expression data.

Methods: This article focuses on familywise error rate (FWER) and false discovery rate (FDR) controlling procedures. Methods under investigation include: Bonferroni-type methods and their improvements (including resampling approaches), modified Bonferroni methods, data-driven approaches, as well as the linear step-up method and its modifications. Particular emphasis lies on the description of the assumptions, advantages and limitations for the investigated methods.

Results: FWER controlling procedures are often too conservative in high dimensional screening studies. A better balance between the raw P-values and the stringent FWER-adjusted P-values may be required in many situations, as provided by FDR controlling and related procedures.

Conclusions: The questions remain open, which error concept to apply and which multiple testing procedure to use. Although we believe that the FDR or one of its variants will be applied more often in the future, longterm experience with microarray technology is missing and thus the validity of appropriate multiple test procedures cannot yet be assessed for microarray data analysis.

Keywords

Familywise error rate - false discovery rate - error concepts - screening
 

  • References

  • 1 Hochberg Y, Tamhane AC. Multiple comparison procedures. New York: Wiley; 1987
    Search in Google Scholar
  • 2 Holm S. A simple sequentially multiple test procedure. Scand J Stat 1979; 6: 65-70.
    PubMedSearch in Google Scholar
  • 3 Shaffer JP. Modified sequentially rejective multiple test procedures. J Am Stat Assoc 1986; 81: 826-31.
    CrossrefPubMedSearch in Google Scholar
  • 4 Westfall PH, Young SS. Resampling-based multiple testing. New York: Wiley; 1993
    Search in Google Scholar
  • 5 Dudoit S, Shaffer JP, Boldrick JC. Multiple hypothesis testing in microarray experiments. Stat Sci 2003; 18: 71-103.
    CrossrefPubMedSearch in Google Scholar
  • 6 Sarkar SK, Chang CK. Simes’ method for multiple hypothesis testing with positively dependent test statistics. J Am Stat Assoc 1997; 92: 1601-8.
    CrossrefPubMedSearch in Google Scholar
  • 7 Simes RJ. An improved Bonferroni procedure for multiple tests of significance. Biometrika 1986; 73: 751-4.
    CrossrefPubMedSearch in Google Scholar
  • 8 Hochberg Y. A sharper Bonferroni procedure for multiple tests of significance. Biometrika 1988; 75: 800-2.
    CrossrefPubMedSearch in Google Scholar
  • 9 Hommel G. A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 1988; 75: 383-6.
    CrossrefPubMedSearch in Google Scholar
  • 10 Westfall PH, Krishen A. Optimally weighted, fixed sequence and gatekeeping multiple testing procedures. J Stat Plan Infer 2001; 99: 25-40.
    CrossrefPubMedSearch in Google Scholar
  • 11 Kropf S, Läuter J. Multiple tests for different sets of variables using a data-driven ordering of hypotheses, with an application to gene expression data. Biom J 2002; 44: 789-800.
    CrossrefPubMedSearch in Google Scholar
  • 12 Läuter J, Glimm E, Kropf S. Multivariate tests based on left-spherically distributed linear scores. Ann Stat 1998; 26: 1972-88.
    CrossrefPubMedSearch in Google Scholar
  • 13 Westfall PH, Kropf S, Finos L. Weighted FWE controlling methods in high-dimensional situations. In Recent developments in multiple comparisons procedures. Benjamini Y, Bretz F, Sarkar S. (eds.) IMS Lecture Notes – Monograph Series, Vol 47. Beachwood, Ohio, USA: 2004: 143-54.
    Search in Google Scholar
  • 14 Hommel G, Kropf S. Tests for Differentiation in Gene Expression Using a Data-Driven Order or Weights for Hypotheses. Biometrical Journal 2005 (to appear).
    PubMed
  • 15 Benjamini Y, Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J Roy Stat Soc B 1995; 57: 289-300.
    PubMedSearch in Google Scholar
  • 16 Seeger P. A note on a method for the analysis of significance en masse. Technometrics 1968; 10: 586-93.
    CrossrefPubMedSearch in Google Scholar
  • 17 Finner H, Roters M. On the false discovery rate and expected type I errors. Biom J 2001; 43: 985-1005.
    CrossrefPubMedSearch in Google Scholar
  • 18 Troendle JF. Stepwise normal theory multiple test procedures controlling the false discovery rate. J Stat Plan Infer 2000; 84: 139-58.
    CrossrefPubMedSearch in Google Scholar
  • 19 Hsu JC, Chang JY, Wang T. Multiple comparisons in screening for differential gene expressions from microarray data. In. Screening Dean A, Lewis S. (eds.) New York: Springer-Verlag; 2004
    Search in Google Scholar
  • 20 Weller JI, Song JZ, Heyen DW, Lewin HA, Ron M. A new approach to the problem of multiple comparisons in the genetic dissection of complex traits. Genetics 1998; 150: 1699-1706.
    PubMedSearch in Google Scholar
  • 21 Zaykin DV, Young SS, Westfall PH. Using false discovery rate approach in the genetic dissection of complex traits: a response to Weller, et al. Genetics 2000; 154: 1917-8.
    PubMedSearch in Google Scholar
  • 22 Storey JD, Taylor JE, Siegmund D. Strong control, conservative point estimation, and simultaneous conservative consistency of false discovery rates: A unified approach. J Roy Stat Soc B 2004; 66: 187-205.
    CrossrefPubMedSearch in Google Scholar
  • 23 Storey JD. A direct approach to false discovery rates. J Roy Stat Soc B 2002; 64: 479-98.
    CrossrefPubMedSearch in Google Scholar
  • 24 Efron B, Tibshirani R, Storey JD, Tusher V. Empirical Bayes analysis of a microarray experiment. J Am Stat Assoc 2001; 96: 1151-60.
    CrossrefPubMedSearch in Google Scholar
  • 25 Korn EL, Troendle JF, McShane LM, Simon R. Controlling the number of false discoveries: Application to high dimensional genomic data. J Stat Plan Infe. 2004. (in press)
    Search in Google Scholar
  • 26 van der Laan MJ, Dudoit S, Pollard KS. Augmentation Procedures for Control of the Generalized Family-Wise Error Rate and Tail Probabilities for the Proportion of False Positives. Statistical Applications in Genetics and Molecular Biology. 2004: 3-15. June
    Search in Google Scholar
  • 27 Hsueh HM, Chen JJ, Kodell RL. Comparison of methods for estimating the number of true null hypotheses in multiplicity testing. J Biopharm Stat 2003; 13: 675-89.
    CrossrefPubMedSearch in Google Scholar
  • 28 Benjamini Y, Hochberg Y. The adaptive control of the false discovery rate in multiple hypotheses testing with independent statistics. J Educ Behav Stat 2000; 25: 60-83.
    CrossrefPubMedSearch in Google Scholar
  • 29 Benjamini Y, Yekutieli D. The control of the false discovery rate in multiple hypothesis testing under dependency. Ann Stat 2001; 29: 1165-88.
    CrossrefPubMedSearch in Google Scholar
  • 30 Yekutieli D, Benjamini Y. Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. J Stat Plan Infer 1999; 82: 171-96.
    CrossrefPubMedSearch in Google Scholar
  • 31 Somerville PN. FDR step-down and step-up procedures for the correlated case. In Recent developments in multiple comparisons procedures.. Benjamini Y, Bretz F, Sarkar S. (eds.) IMS Lecture Notes – Monograph Series, Vol. 47. Beachwood, Ohio, USA: 2004: 100-18.
    Search in Google Scholar
  • 32 Benjamini Y, Liu W. A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence. J Stat Plan Infer 1999; 82: 163-70.
    CrossrefPubMedSearch in Google Scholar
  • 33 Horn M, Dunnett CW. Power and sample size comparisons of stepwise FWE and FDR controlling test procedures in the normal many-one case. In Recent developments in multiple comparisons procedures. Benjamini Y, Bretz F, Sarkar S. (eds.) IMS Lecture Notes – Monograph Series, Vol. 47. Beachwood, Ohio, USA: 2004: 48-64.
    Search in Google Scholar
  • 34 Dudoit S, van der Laan MJ, Pollard KS. Multiple testing. Part I. Single-step procedures for control of general type I error rates. Statistical Applications in Genetics and Molecular Biology 2004; 3 June 14
    PubMedSearch in Google Scholar
  • 35 van der Laan MJ, Dudoit S, Pollard KS. Multiple testing. Part II. Step-down procedures for control of the family-wise error rate. Statistical Applications in Genetics and Molecular Biology 2004; 3 June 9
    PubMedSearch in Google Scholar
  • 36 Golub TR, Slonim DK, Tamayo P, Huard C, Gaasenbeek M, Mesirov JP, Coller H, Loh ML, Downing JR, Caligiuri MA, Bloomfield CD, Lander ES. Molecular classification of cancer: Class discovery and class prediction by gene expression monitoring. Science 1999; 286: 531-7.
    CrossrefPubMedSearch in Google Scholar
  • 37 Tusher VG, Tibshirani R, Chu G. Significance analysis of microarrays applied to ionizing radiation response. Proc Ntl Acad Sci 2001; 98: 5116-21.
    CrossrefPubMedSearch in Google Scholar
  • 38 Ge Y, Dudoit S, Speed T. Resampling-based multiple testing for microarray data analysis. Test 2003; 12: 1-77.
    CrossrefPubMedSearch in Google Scholar