Simple solution to a common statistical problem: Interpreting multiple tests
91
Citation
14
Reference
10
Related Paper
Citation Trend
Keywords:
Bonferroni correction
Multiple comparisons problem
p-value
False Discovery Rate
Nominal level
Statistical power
Results on the false discovery rate (FDR) and the false nondiscovery rate (FNR) are developed for single-step multiple testing procedures. In addition to verifying desirable properties of FDR and FNR as measures of error rates, these results extend previously known results, providing further insights, particularly under dependence, into the notions of FDR and FNR and related measures. First, considering fixed configurations of true and false null hypotheses, inequalities are obtained to explain how an FDR- or FNR-controlling single-step procedure, such as a Bonferroni or Šidák procedure, can potentially be improved. Two families of procedures are then constructed, one that modifies the FDR-controlling and the other that modifies the FNR-controlling Šidák procedure. These are proved to control FDR or FNR under independence less conservatively than the corresponding families that modify the FDR- or FNR-controlling Bonferroni procedure. Results of numerical investigations of the performance of the modified Šidák FDR procedure over its competitors are presented. Second, considering a mixture model where different configurations of true and false null hypotheses are assumed to have certain probabilities, results are also derived that extend some of Storey's work to the dependence case.
False Discovery Rate
Multiple comparisons problem
Cite
Citations (80)
동시에 여러 개의 가설검정 수행시 귀무가설이 참일 경우 귀무가설을 기각할 확률이 커지는 문제가 발생한다. 이러한 다중검정 문제 해결을 위해 여러 연구에서는 가설검정시 필요한 집단별 오류율(FWER; family-wise error rate), 위발견율 (FDR; false discovery rate) 또는 위비발견율 (FNR; false nondiscovery rate) 과 통계량을 고려하여 검정력을 높이고자 하였다. 본 연구에서는 T 통계량, 수정된 T 통계량, 그리고 LPE (local pooled error) 통계량 기반 P값을 이용한 Bonferroni (1960) 방법, Holm (1979) 방법, Benjamini와 Hochberg (1995) 방법과 Benjamini와 Yekutieli (2001) 방법 그리고 Z 통계량 기반 Sun과 Cai (2007) 방법을 고찰하고 모의실험을 통해 다중검정 능력을 비교하였다. 또한 실제 데이터로 애기장대 유전자 발현 데이터에 대해 여러 가지 다중검정법을 통해 유의한 유전자들을 선별하였다. When thousands of hypotheses are tested simultaneously, the probability of rejecting any true hypotheses increases, and large multiplicity problems are generated. To solve these problems, researchers have proposed different approaches to multiple testing methods, considering family-wise error rate (FWER), false discovery rate (FDR) or false nondiscovery rate (FNR) as a type I error and some test statistics. In this article, we discuss Bonferroni (1960), Holm (1979), Benjamini and Hochberg (1995) and Benjamini and Yekutieli (2001) procedures based on T statistics, modified T statistics or local-pooled-error (LPE) statistics. We also consider Sun and Cai (2007) procedure based on Z statistics. These procedures are compared in the simulation and applied to Arabidopsis microarray gene expression data to identify differentially expressed genes.
False Discovery Rate
Bonferroni correction
Multiple comparisons problem
Word error rate
False positive rate
Cite
Citations (1)
This paper is a review of the popular Benjamini Hochberg Method and other related useful methods of Multiple Hypothesis testing. This is written with the purpose of serving a short but complete easy to understand review of the main article with proper background. The paper titled 'Controlling the False Discovery Rate-a practical and powerful Approach to multiple Testing' by benjamini et. al.[1] proposes a new framework of controlling the False Discovery Rate in a Multiple Hypothesis testing problem. It has been claimed that the procedure proposed in the paper results in a substantial gain in power more applicable in case of problems which call for False discovery rate (FDR) control rather than Familywise Error Rate (FWER). The proposed method uses a simple Bonferroni type procedure for FDR control.
False Discovery Rate
Bonferroni correction
Multiple comparisons problem
Word error rate
Cite
Citations (0)
False Discovery Rate
Bonferroni correction
Multiple comparisons problem
Corollary
Nominal level
Cite
Citations (5)
Bonferroni correction
False Discovery Rate
Multiple comparisons problem
Cite
Citations (1,219)
It is a typical feature of high dimensional data analysis, for example a microarray study, that a researcher allows thousands of statistical tests at a time. All inferences for the tests are determined using the p-values; a smaller p-value than the α-level of the test signifies a statistically significant test. As the number of tests increases, the chance of observing some small p-values is very high even when all null hypotheses are true. Consequently, we make wrong conclusions on the hypotheses. This type of potential problem frequently happens when we test several hypotheses simultaneously, i.e., the multiple testing problem. Adjustment of the p-values can redress the problem that arises in multiple hypothesis testing. P-value adjustment methods control error rates [type I error (i.e. false positive) and type II error (i.e. false negative)] for each hypothesis in order to achieve high statistical power while keeping the overall Family Wise Error Rate (FWER) no larger than α, where α is most often set to 0.05. However, researchers also consider the False Discovery Rate (FDR), or Positive False Discovery Rate (pFDR) instead of the type I error in multiple comparison problems for microarray studies. The methods involved in controlling the FDR always provide higher statistical power than the methods involved in controlling the type I error rate while keeping the type II error rate low. In practice, microarray studies involve dependent test statistics (or p-values) because genes can be fully dependent on each other in a complicated biological structure. However, some of the p-value adjustment methods only deal with independent test statistics. Thus, we carry out a simulation study with several methods involved in multiple hypothesis testing.
False Discovery Rate
Multiple comparisons problem
p-value
Statistical power
Word error rate
False positive rate
Nominal level
Cite
Citations (0)
Multiple tests are designed to test a whole collection of null hypotheses simultaneously. Their quality is often judged by the false discovery rate (FDR), i.e. the expectation of the quotient of the number of false rejections divided by the amount of all rejections. The widely cited Benjamini and Hochberg (BH) step up multiple test controls the FDR under various regularity assumptions. In this note we present a rapid approach to the BH step up and step down tests. Also sharp FDR inequalities are discussed for dependent p-values and examples and counter-examples are considered. In particular, the Bonferroni bound is sharp under dependence for control of the family-wise error rate.
False Discovery Rate
Bonferroni correction
Multiple comparisons problem
Null (SQL)
Cite
Citations (0)
Multiple tests are designed to test a whole collection of null hypotheses simultaneously. Their quality is often judged by the false discovery rate (FDR), i.e. the expectation of the quotient of the number of false rejections divided by the amount of all rejections. The widely cited Benjamini and Hochberg (BH) step up multiple test controls the FDR under various regularity assumptions. In this note we present a rapid approach to the BH step up and step down tests. Also sharp FDR inequalities are discussed for dependent p-values and examples and counter-examples are considered. In particular, the Bonferroni bound is sharp under dependence for control of the family-wise error rate.
False Discovery Rate
Bonferroni correction
Multiple comparisons problem
Null (SQL)
Word error rate
Cite
Citations (0)
This paper is a review of the popular Benjamini Hochberg Method and other related useful methods of Multiple Hypothesis testing. This is written with the purpose of serving a short but complete easy to understand review of the main article with proper background. The paper titled 'Controlling the False Discovery Rate-a practical and powerful Approach to multiple Testing' by benjamini et. al.[1] proposes a new framework of controlling the False Discovery Rate in a Multiple Hypothesis testing problem. It has been claimed that the procedure proposed in the paper results in a substantial gain in power more applicable in case of problems which call for False discovery rate (FDR) control rather than Familywise Error Rate (FWER). The proposed method uses a simple Bonferroni type procedure for FDR control.
False Discovery Rate
Multiple comparisons problem
Bonferroni correction
Word error rate
Cite
Citations (1)
동시에 여러 개의 가설검정 수행시 귀무가설이 참일 경우 귀무가설을 기각할 확률이 커지는 문제가 발생한다. 이러한 다중검정 문제 해결을 위해 여러 연구에서는 가설검정시 필요한 집단별 오류율(FWER; family-wise error rate), 위발견율 (FDR; false discovery rate) 또는 위비발견율 (FNR; false nondiscovery rate) 과 통계량을 고려하여 검정력을 높이고자 하였다. 본 연구에서는 T 통계량, 수정된 T 통계량, 그리고 LPE (local pooled error) 통계량 기반 P값을 이용한 Bonferroni (1960) 방법, Holm (1979) 방법, Benjamini와 Hochberg (1995) 방법과 Benjamini와 Yekutieli (2001) 방법 그리고 Z 통계량 기반 Sun과 Cai (2007) 방법을 고찰하고 모의실험을 통해 다중검정 능력을 비교하였다. 또한 실제 데이터로 애기장대 유전자 발현 데이터에 대해 여러 가지 다중검정법을 통해 유의한 유전자들을 선별하였다. 【When thousands of hypotheses are tested simultaneously, the probability of rejecting any true hypotheses increases, and large multiplicity problems are generated. To solve these problems, researchers have proposed different approaches to multiple testing methods, considering family-wise error rate (FWER), false discovery rate (FDR) or false nondiscovery rate (FNR) as a type I error and some test statistics. In this article, we discuss Bonferroni (1960), Holm (1979), Benjamini and Hochberg (1995) and Benjamini and Yekutieli (2001) procedures based on T statistics, modified T statistics or local-pooled-error (LPE) statistics. We also consider Sun and Cai (2007) procedure based on Z statistics. These procedures are compared in the simulation and applied to Arabidopsis microarray gene expression data to identify differentially expressed genes.】
False Discovery Rate
Bonferroni correction
Multiple comparisons problem
Word error rate
Cite
Citations (0)