Family-wise error rate | EPFL Graph Search

In statistics, family-wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors when performing multiple hypotheses tests. Familywise and Experimentwise Error Rates John Tukey developed in 1953 the concept of a familywise error rate as the probability of making a Type I error among a specified group, or "family," of tests. Ryan (1959) proposed the related concept of an experimentwise error rate, which is the probability of making a Type I error in a given experiment. Hence, an experimentwise error rate is a familywise error rate for all of the tests that are conducted within an experiment. As Ryan (1959, Footnote 3) explained, an experiment may contain two or more families of multiple comparisons, each of which relates to a particular statistical inference and each of which has its own separate familywise error rate. Hence, familywise error rates are usually based on theoretically informative collections of

Some Thoughts on Robustness and Multiple Testing

Djalel Eddine Meskaldji, Stephan Morgenthaler

When testing many null hypotheses, deciding which of them to reject is a subtle game. The prevailing approach consists in deciding on a level and type of control against false rejections (errors of type I) and subject to this constraint to maximize the number of rejections. The two extreme types of control are the FWER (family-wise error rate, probability of at least one false rejection) and FDR (false discovery rate, expected rate of false rejections among all rejections). In this talk, I will discuss two topics. First, how to construct alternatives to these two procedures together with elements of an optimality theory and, second, some considerations about robustness. The FWER and the FDR detect alternatives by comparing the ordered p-values to a boundary; a constant equal to =m for FWER and a boundary that is linearly growing in the rank r equal to r=m for FDR. We will show that if the boundary is equal to s(r)=m, a certain control is implied. By choosing Huber’s clipped linear function s(r) = min(k; r) a family of multiple tests, bridging FWER and FDR is created. How to choose k will also be discussed. The theory of control is based on the fact that the p-values for the true hypotheses are uniformly distributed. If the test is approximate and the nomial p-value is not equal to the real one, this no longer holds. In this case, the control is also merely nominal. We will discuss some examples to illustrate this point and to measure the likely effects on the conclusions of a multiple testing procedure.

2012

Comparing multiple testing procedures and error rates under weighted classification

Djalel Eddine Meskaldji, Stephan Morgenthaler, Dimitri Nestor Alice Van De Ville

When many hypotheses are tested simultaneously, the control of the number of false rejections is often the principal consideration. In practice, this is insatisfacory since we also have to keep in mind the power of detecting true effects. At the moment, the preferred choice of practitioners is often restricted to family-wise error rates (FWER) and false discovery rates (FDR). We recently introduced the scaled multiple testing error rates, which includes most existing error rates and bridges the gap between the FWER and the FDR. For example, the Scaled False Discovery Rate (SFDR) limits the number of false positives (FP) relative to an arbitrary increasing function (s) of the number of rejections (R), by bounding E(FP/s(R)). We compare the performance for different choices of the scaling function s and discuss the optimality of the error rates in some practical scenarios by considering the number of false positives FP separately from the number of true positives TP.

2014

A comprehensive error rate for multiple testing

Djalel Eddine Meskaldji, Stephan Morgenthaler, Jean-Philippe Thiran

In multiple testing, a variety of control metrics have been introduced such as the family-wise error rate (FWER), the False Discovery Rate (FDR), False Exceedence Rate (FER), etc. We found a way to embed these metrics into a continuous family of control metrics, all of which can be attained by applying a simple and general family of multiple testing procedures. The new general error rate (GER) limits the number of false positives relative to an arbitrary increasing function of the number of rejections. An example is

FR/R^\gamma

, the number of false rejections divided by the number of rejections to a power

0\leq \gamma\leq 1

. We investigated both the control of quantiles and of expectations and provide the corresponding multiple testing procedures. In the above example, the expectation of the criterion thus leads to a family of multiple testing procedures that bridges the gap between FWER and FDR.

2011

A comprehensive error rate for multiple testing

Djalel Eddine Meskaldji, Stephan Morgenthaler, Jean-Philippe Thiran

FR/R^\gamma

, the number of false rejections divided by the number of rejections to a power

0\leq \gamma\leq 1

2011