Analysis of a simulated microarray dataset: Comparison of methods for data normalisation and detection of differential expression (Open Access publication)
Mick WatsonMónica Pérez-AlegreMichael D. BaronCéline DelmasPeter DovčMylène DuvalJean‐Louis FoulleyJuan J. GarridoIna HulseggeFlorence JaffrézicÁngeles Jiménez-MarínMiha LavričKim‐Anh Lê CaoGuillemette MarotDaphné MouzakiM.H. PoolChristian P. RobertMagali San CristobalGwenola Tosser‐KloppDavid WaddingtonDirk‐Jan de Koning
6
Citation
19
Reference
10
Related Paper
Abstract:
Abstract Microarrays allow researchers to measure the expression of thousands of genes in a single experiment. Before statistical comparisons can be made, the data must be assessed for quality and normalisation procedures must be applied, of which many have been proposed. Methods of comparing the normalised data are also abundant, and no clear consensus has yet been reached. The purpose of this paper was to compare those methods used by the EADGENE network on a very noisy simulated data set. With the a priori knowledge of which genes are differentially expressed, it is possible to compare the success of each approach quantitatively. Use of an intensity-dependent normalisation procedure was common, as was correction for multiple testing. Most variety in performance resulted from differing approaches to data quality and the use of different statistical tests. Very few of the methods used any kind of background correction. A number of approaches achieved a success rate of 95% or above, with relatively small numbers of false positives and negatives. Applying stringent spot selection criteria and elimination of data did not improve the false positive rate and greatly increased the false negative rate. However, most approaches performed well, and it is encouraging that widely available techniques can achieve such good results on a very noisy data set.Keywords:
False Discovery Rate
False positive rate
Multiple comparisons problem
Data set
Abstract Genome-Wide Association Studies are an important tool for identifying genetic markers associated with a trait, but it has been plagued by the multiple testing problem, which necessitates a multiple testing correction method. While many multiple testing methods have been suggested, e.g. Bonferroni and Benjamini-Hochberg’s False Discovery Rate, the quality of the adjusted threshold based on these methods is not as well investigated. The aim of this study was to evaluate the balance between power and false positive rate of a Genome-Wide Association Studies experiment with Bonferroni and Benjamini-Hochberg’s False Discovery Rate multiple testing correction methods and to test the effects of various experimental design and genetic architecture parameters on this balance. Our results suggest that when the markers are independent the threshold from Benjamini-Hochberg’s False Discovery Rate provides a better balance between power and false positive rate in an experiment. However, with correlations between markers the threshold of Benjamini-Hochberg’s False Discovery Rate becomes too lenient with an excessive number of false positives. Experimental design parameters such as sample size and number of markers used, as well as genetic architecture of a trait affect the balance between power and false positive rate. This experiment provided guidance in selecting an appropriate experimental design and multiple testing correction method when conducting an experiment.
False Discovery Rate
Bonferroni correction
Multiple comparisons problem
False positive rate
Trait
Statistical power
Genetic architecture
Cite
Citations (1)
Multiple testing procedures are commonly used in gene expression studies for the detection of differential expression, where typically thousands of genes are measured over at least two experimental conditions. Given the need for powerful testing procedures, and the attendant danger of false positives in multiple testing, the False Discovery Rate (FDR) controlling procedure of Benjamini and Hochberg (1995) has become a popular tool. When simultaneously testing hypotheses, suppose that R rejections are made, of which Fp are false positives. The Benjamini and Hochberg procedure ensures that the expectation of Fp/R is bounded above by some pre-specified proportion. In practice, the procedure is applied to a single experiment. In this paper we investigate the across-experiment variability of the proportion Fp/R as a function of three experimental parameters. The operational characteristics of the procedure when applied to dependent hypotheses are also considered.
False Discovery Rate
Multiple comparisons problem
Expression (computer science)
True positive rate
False positive rate
Cite
Citations (118)
False positives are equally dangerous as false negatives. Ideally the false positive rate should remain 0 or very close to 0. Even a slightest increase in false positive rate is considered as undesirable.
Although the specific methods provide very accurate scanning by comparing viruses with their exact signatures, they fail to detect the new and unknown viruses. On the other hand the generic methods can detect even new viruses without using virus signatures. But these methods are more likely to generate false positives. There is a positive correlation between the capability to detect new and unknown viruses and false positive rate.
While a traditional approach tries to achieve a right balance between false positives and false negatives a TRIZ approach looks forward to achieve the Ideal Final Result. The Ideal final result is to 'detect and prevent viruses with full certainty. The chances of error should be nil and the method should not raise any false positive or false negative.' The article shows many contradictions relating to false positives and their solutions.
False positive rate
True positive rate
False Negative Reactions
Cite
Citations (0)
In multiple testing, a variety of control metrics of false positives have been introduced such as the Per Family Error Rate (PFER), Family-Wise Error Rate (FWER), the False Discovery Rate (FDR), the False Exceedence Rate (FER). In this talk, we present a comprehensive family of error rates together with a corresponding family of multiple testing procedures (MTP). Based on the needs of the problem at hand, the user can choose a particular member among these MTPs. The new error rate limits the number of false positives FP relative to an arbitrary non-decreasing function s of the number of rejections R. The quantity is called, the scaled false discovery proportion SFDP=FP/s(R). We present different procedures to control either the P(SFDP>q) or the E(SFDP) for any choice of the scaling function. An obvious choice is s(R)=min(R;k). As does FDR, this particular error rate FP/s(R) accepts a fixed percentage of false rejections among all rejections, but only up to R =k, then a stricter control takes over and for R > k, the number of false positives is limited to a percentage of the fixed value k, similar to PFER. The corresponding family of multiple testing procedures bridges the gap between the PFER (k =1) and the FDR (k = number of tests). A similar such bridge is obtained when s(R)=Rg with 0<=g<=1, which for g=0.5 controls the percentage of false discoveries relative to the square root of R. In the talk, we discuss the choice of the parameters k and g based on the minimization of the expected loss t E(FP) - E(TP) = t E(FP) - E(R - FP) which is based on the idea that a false positive costs a penalty of 1
False Discovery Rate
Multiple comparisons problem
Word error rate
False positive rate
Cite
Citations (0)
Abstract Genome scan mapping experiments involve multiple tests of significance. Thus, controlling the error rate in such experiments is important. Simple extension of classical concepts results in attempts to control the genomewise error rate (GWER), i.e., the probability of even a single false positive among all tests. This results in very stringent comparisonwise error rates (CWER) and, consequently, low experimental power. We here present an approach based on controlling the proportion of false positives (PFP) among all positive test results. The CWER needed to attain a desired PFP level does not depend on the correlation among the tests or on the number of tests as in other approaches. To estimate the PFP it is necessary to estimate the proportion of true null hypotheses. Here we show how this can be estimated directly from experimental results. The PFP approach is similar to the false discovery rate (FDR) and positive false discovery rate (pFDR) approaches. For a fixed CWER, we have estimated PFP, FDR, pFDR, and GWER through simulation under a variety of models to illustrate practical and philosophical similarities and differences among the methods.
False Discovery Rate
Multiple comparisons problem
False positive rate
Statistical power
Word error rate
True positive rate
Cite
Citations (167)
False Discovery Rate
Multiple comparisons problem
False positive rate
p-value
Value (mathematics)
Cite
Citations (9)
False positives are equally dangerous as false negatives. Ideally the false positive rate should remain 0 or very close to 0. Even a slightest increase in false positive rate is considered as undesirable.
Although the specific methods provide very accurate scanning by comparing viruses with their exact signatures, they fail to detect the new and unknown viruses. On the other hand the generic methods can detect even new viruses without using virus signatures. But these methods are more likely to generate false positives. There is a positive correlation between the capability to detect new and unknown viruses and false positive rate.
While a traditional approach tries to achieve a right balance between false positives and false negatives a TRIZ approach looks forward to achieve the Ideal Final Result. The Ideal final result is to 'detect and prevent viruses with full certainty. The chances of error should be nil and the method should not raise any false positive or false negative.' The article shows many contradictions relating to false positives and their solutions.
False positive rate
True positive rate
False Negative Reactions
Cite
Citations (0)
False positives are equally dangerous as false negatives. Ideally the false positive rate should remain 0 or very close to 0. Even a slightest increase in false positive rate is considered as undesirable. Although the specific methods provide very accurate scanning by comparing viruses with their exact signatures, they fail to detect the new and unknown viruses. On the other hand the generic methods can detect even new viruses without using virus signatures. But these methods are more likely to generate false positives. There is a positive correlation between the capability to detect new and unknown viruses and false positive rate. While a traditional approach tries to achieve a right balance between false positives and false negatives a TRIZ approach looks forward to achieve the Ideal Final Result. The Ideal final result is to 'detect and prevent viruses with full certainty. The chances of error should be nil and the method should not raise any false positive or false negative.' The article shows many contradictions relating to false positives and their solutions.
False positive rate
True positive rate
False Negative Reactions
Cite
Citations (0)
1. SUMMARY In a multiple testing setting, the investigator is faced with choosing what error to control and what method to use in controlling that error. Considerations include the assumptions that are made for each method and if the assumptions are acceptable in that setting. Failure to acknowledge these considerations can lead to drastically misleading results. Our recent study showed that, in applications where reduced multiplicity is encountered, the widely used Benjamini and Hochberg’s (BH) false discovery rate (FDR) analysis is less robust than approaches controlling the number of false positives. In this manuscript we assess the current methods to control the probability of committing a flxed number of false positives and provide a new method. We provide theoretical proof that our proposed approach, KBIN, is more powerful than alternative approaches. We also conduct simulations and real data studies to evaluate the proposed flnding. We expect that the KBIN method has promising applications in biomarker settings where the goal is to choose a set of signiflcant biomarkers from among a panel of potential putative biomarkers.
False Discovery Rate
Multiple comparisons problem
Word error rate
Cite
Citations (1)
An incredible amount of data is generated in the course of a functional neuroimaging experiment. The quantity of data gives us improved temporal and spatial resolution with which to evaluate our results. It also creates a staggering multiple testing problem. A number of methods have been created that address the multiple testing problem in neuroimaging in a principled fashion. These methods place limits on either the familywise error rate (FWER) or the false discovery rate (FDR) of the results. These principled approaches are well established in the literature and are known to properly limit the amount of false positives across the whole brain. However, a minority of papers are still published every month using methods that are improperly corrected for the number of tests conducted. These latter methods place limits on the voxelwise probability of a false positive and yield no information on the global rate of false positives in the results. In this commentary, we argue in favor of a principled approach to the multiple testing problem—one that places appropriate limits on the rate of false positives across the whole brain gives readers the information they need to properly evaluate the results.
False Discovery Rate
Multiple comparisons problem
True positive rate
False positive rate
Cite
Citations (457)