Exact test

In statistics, an exact (significance) test is a test where if the Null hypotheses is true then all assumptions, upon which the derivation of the distribution of the test statistic is based, are met. Using an exact test provides a significance test that keeps the Type I error rate of the test ( α {displaystyle alpha } ) at the desired significance level of the test. For example an exact test at significance level of α = 5 % {displaystyle alpha =5\%} , when repeating the test over many samples where the null hypotheses is true, will reject at most 5 % {displaystyle 5\%} of the time. This is opposed to an approximate test in which the desired type I error rate is only approximately kept (i.e.: the test might reject more than 5% of the time), while this approximation may be made as close to α {displaystyle alpha } as desired by making the sample size big enough. In statistics, an exact (significance) test is a test where if the Null hypotheses is true then all assumptions, upon which the derivation of the distribution of the test statistic is based, are met. Using an exact test provides a significance test that keeps the Type I error rate of the test ( α {displaystyle alpha } ) at the desired significance level of the test. For example an exact test at significance level of α = 5 % {displaystyle alpha =5\%} , when repeating the test over many samples where the null hypotheses is true, will reject at most 5 % {displaystyle 5\%} of the time. This is opposed to an approximate test in which the desired type I error rate is only approximately kept (i.e.: the test might reject more than 5% of the time), while this approximation may be made as close to α {displaystyle alpha } as desired by making the sample size big enough. Exact tests that are based on discrete test statistic may be conservative tests, i.e. that its actual rejection rate is below the nominal significance level α {displaystyle alpha } . For example, this is the case for Fisher's exact test. If the test statistic is continuous, it will reach the significance level exactly. Parametric tests, such as those described in exact statistics, are exact tests when the parametric assumptions are fully met, but in practice the use of the term exact (significance) test is reserved for those tests that do not rest on parametric assumptions – non-parametric tests. However, in practice most implementations of non-parametric test software use asymptotical algorithms for obtaining the significance value, which makes the implementation of the test non-exact. So when the result of a statistical analysis is said to be an “exact test” or an “exact p-value”, it ought to imply that the test is defined without parametric assumptions and evaluated without using approximate algorithms. In principle however it could also mean that a parametric test has been employed in a situation where all parametric assumptions are fully met, but it is in most cases impossible to prove this completely in a real world situation. Exceptions when it is certain that parametric tests are exact include tests based on the binomial or Poisson distributions. Sometimes permutation test is used as a synonym for exact test, but although all permutation tests are exact tests, not all exact tests are permutation tests.