How well do multiple testing methods scale up when both n and k increase?

J Biopharm Stat. 2011 Jul;21(4):583-94. doi: 10.1080/10543406.2011.551326.

Abstract

With increasingly massive data sets in biopharmaceutical research, particularly in genomic and related applications, there is concern about how well multiple comparisons methods "scale up" with increasing number of tests (k). Familywise error rate-controlling methods do not scale up well, and false discovery rate-controlling methods do scale up well with increasing k. But neither method scales up well with increasing sample size (n) when testing point nulls. We develop a loss function approach to investigate scale-up properties of various methods; we find that while Efron's recent proposal scales up best when both sample size n and number of tests k increase, but its performance otherwise can be erratic.

MeSH terms

  • Biopharmaceutics* / methods
  • Biopharmaceutics* / statistics & numerical data
  • Biostatistics / methods*
  • Clinical Trials as Topic / statistics & numerical data
  • Computer Simulation
  • Data Interpretation, Statistical*
  • Endpoint Determination
  • Gene Expression Profiling / statistics & numerical data
  • Models, Statistical*
  • Sample Size