A weighted FDR procedure under discrete and heterogeneous null distributions
7
Citation
37
Reference
10
Related Paper
Citation Trend
Abstract:
Abstract Multiple testing (MT) with false discovery rate (FDR) control has been widely conducted in the “discrete paradigm” where p ‐values have discrete and heterogeneous null distributions. However, in this scenario existing FDR procedures often lose some power and may yield unreliable inference, and for this scenario there does not seem to be an FDR procedure that partitions hypotheses into groups, employs data‐adaptive weights and is nonasymptotically conservative. We propose a weighted p ‐value‐based FDR procedure, “weighted FDR (wFDR) procedure” for short, for MT in the discrete paradigm that efficiently adapts to both heterogeneity and discreteness of p ‐value distributions. We theoretically justify the nonasymptotic conservativeness of the wFDR procedure under independence, and show via simulation studies that, for MT based on p ‐values of binomial test or Fisher's exact test, it is more powerful than six other procedures. The wFDR procedure is applied to two examples based on discrete data, a drug safety study, and a differential methylation study, where it makes more discoveries than two existing methods.Keywords:
False Discovery Rate
Multiple comparisons problem
Independence
Nominal level
p-value
A useful paradigm for multiple testing is to control error rates derived from the false discovery proportion (FDP). The false discovery rate (FDR) is the expectation of the FDP, which is defined to be zero if no rejection is made. How- ever, since follow-up studies are based on hypotheses that are actually rejected, it is important to control the positive FDR (pFDR) or the positive false discovery excessive probability (pFDEP), i.e., the conditional expectation of the FDP or the conditional probability of the FDP exceeding a specified level, given that at least one rejection is made. We show that, unlike FDR, these two positive error rates may not be controllable at a desired level. Given a multiple testing problem, there can exist positive intrinsic lower bounds, such that no procedures can attain a pFDR or pFDEP level below the corresponding bound. To reduce misinterpretations of testing results, we propose several procedures that are adaptive, i.e., they achieve pFDR or pFDEP control when the target control level is attainable, and make no rejections otherwise. The adaptive control is established under a sparsity condition where the fraction of false nulls is increasingly close to zero as well as under the con- dition where the fraction of false nulls is a positive constant. We demonstrate that the power of the proposed procedures is comparable to the Benjamini-Hochberg FDR controlling procedure.
False Discovery Rate
Multiple comparisons problem
Constant (computer programming)
Fraction (chemistry)
Nominal level
False positive rate
Probability of error
Positive control
Cite
Citations (17)
False Discovery Rate
Multiple comparisons problem
Unit interval
p-value
Value (mathematics)
Null (SQL)
Cite
Citations (3)
False Discovery Rate
Multiple comparisons problem
p-value
Nuisance parameter
Null (SQL)
Cite
Citations (9)
False Discovery Rate
Bonferroni correction
Multiple comparisons problem
Corollary
Nominal level
Cite
Citations (5)
When many tests of significance are examined in a research investigation with procedures that limit the probability of making at least one Type I error--the so-called familywise techniques of control--the likelihood of detecting effects can be very low. That is, when familywise error controlling methods are adopted to assess statistical significance, the size of the critical value that must be exceeded in order to obtain statistical significance can be extremely large when the number of tests to be examined is also very large. In our investigation we examined three methods for increasing the sensitivity to detect effects when family size is large: the false discovery rate of error control presented by Benjamini and Hochberg (1995), a modified false discovery rate presented by Benjamini and Hochberg (2000) which estimates the number of true null hypotheses prior to adopting false discovery rate control, and a familywise method modified to control the probability of committing two or more Type I errors in the family of tests examined--not one, as is the case with the usual familywise techniques. Our results indicated that the level of significance for the two or more familywise method of Type I error control varied with the testing scenario and needed to be set on occasion at values in excess of 0.15 in order to control the two or more rate at a reasonable value of 0.01. In addition, the false discovery rate methods typically resulted in substantially greater power to detect non-null effects even though their levels of significance were set at the standard 0.05 value. Accordingly, we recommend the Benjamini and Hochberg (1995, 2000) methods of Type I error control when the number of tests in the family is large.
False Discovery Rate
Multiple comparisons problem
Statistical power
p-value
Word error rate
Null (SQL)
Statistical Process Control
False positive rate
Cite
Citations (157)
It is a typical feature of high dimensional data analysis, for example a microarray study, that a researcher allows thousands of statistical tests at a time. All inferences for the tests are determined using the p-values; a smaller p-value than the α-level of the test signifies a statistically significant test. As the number of tests increases, the chance of observing some small p-values is very high even when all null hypotheses are true. Consequently, we make wrong conclusions on the hypotheses. This type of potential problem frequently happens when we test several hypotheses simultaneously, i.e., the multiple testing problem. Adjustment of the p-values can redress the problem that arises in multiple hypothesis testing. P-value adjustment methods control error rates [type I error (i.e. false positive) and type II error (i.e. false negative)] for each hypothesis in order to achieve high statistical power while keeping the overall Family Wise Error Rate (FWER) no larger than α, where α is most often set to 0.05. However, researchers also consider the False Discovery Rate (FDR), or Positive False Discovery Rate (pFDR) instead of the type I error in multiple comparison problems for microarray studies. The methods involved in controlling the FDR always provide higher statistical power than the methods involved in controlling the type I error rate while keeping the type II error rate low. In practice, microarray studies involve dependent test statistics (or p-values) because genes can be fully dependent on each other in a complicated biological structure. However, some of the p-value adjustment methods only deal with independent test statistics. Thus, we carry out a simulation study with several methods involved in multiple hypothesis testing.
False Discovery Rate
Multiple comparisons problem
p-value
Statistical power
Word error rate
False positive rate
Nominal level
Cite
Citations (0)
In experimental research, a statistical test is often used for making decisions on a null hypothesis such as that the means of gene expression in the normal and tumor groups are equal. Typically, a test statistic and its corresponding P value are calculated to measure the extent of the difference between the two groups. The null hypothesis is rejected and a discovery is declared when the P value is less than a prespecified significance level. When more than one test is conducted, use of a significance level intended for use by a single test typically leads to a large chance of false-positive findings.This paper presents an overview of the multiple testing framework and describes the false discovery rate (FDR) approach to determining the significance cutoff when a large number of tests are conducted.The FDR is the expected proportion of the null hypotheses that are falsely rejected divided by the total number of rejections. An FDR-controlling procedure is described and illustrated with a numerical example.In multiple testing, a classical "family-wise error rate" (FWE) approach is commonly used when the number of tests is small. When a study involves a large number of tests, the FDR error measure is a more useful approach to determining a significance cutoff, as the FWE approach is too stringent. The FDR approach allows more claims of significant differences to be made, provided the investigator is willing to accept a small fraction of false-positive findings.
False Discovery Rate
Cut-off
Multiple comparisons problem
p-value
Statistic
Null (SQL)
Value (mathematics)
Statistical power
Cite
Citations (58)
Summary Multiple-hypothesis testing involves guarding against much more complicated errors than single-hypothesis testing. Whereas we typically control the type I error rate for a single-hypothesis test, a compound error rate is controlled for multiple-hypothesis tests. For example, controlling the false discovery rate FDR traditionally involves intricate sequential p-value rejection methods based on the observed data. Whereas a sequential p-value method fixes the error rate and estimates its corresponding rejection region, we propose the opposite approach—we fix the rejection region and then estimate its corresponding error rate. This new approach offers increased applicability, accuracy and power. We apply the methodology to both the positive false discovery rate pFDR and FDR, and provide evidence for its benefits. It is shown that pFDR is probably the quantity of interest over FDR. Also discussed is the calculation of the q-value, the pFDR analogue of the p-value, which eliminates the need to set the error rate beforehand as is traditionally done. Some simple numerical examples are presented that show that this new approach can yield an increase of over eight times in power compared with the Benjamini–Hochberg FDR method.
False Discovery Rate
Multiple comparisons problem
Word error rate
Value (mathematics)
p-value
Statistical power
False positive rate
Cite
Citations (5,479)
False Discovery Rate
Multiple comparisons problem
Nuisance parameter
p-value
Value (mathematics)
Cite
Citations (2)
Multiple testing adjustments, such as the Benjamini and Hochberg (1995) step-up procedure for controlling the false discovery rate (FDR), are typically applied to families of tests that control significance level in the classical sense: for each individual test, the probability of false rejection is no greater than the nominal level. In this paper, we consider tests that satisfy only a weaker notion of significance level control, in which the probability of false rejection need only be controlled on average over the hypotheses. We find that the Benjamini and Hochberg (1995) step-up procedure still controls FDR in the asymptotic regime with many weakly dependent $p$-values, and that certain adjustments for dependent $p$-values such as the Benjamini and Yekutieli (2001) procedure continue to yield FDR control in finite samples. Our results open the door to FDR controlling procedures in nonparametric and high dimensional settings where weakening the notion of inference allows for large power improvements.
False Discovery Rate
Multiple comparisons problem
Nominal level
Statistical Inference
Cite
Citations (0)