Multiple comparisons problem | EPFL Graph Search

In statistics, the multiple comparisons, multiplicity or multiple testing problem occurs when one considers a set of statistical inferences simultaneously or infers a subset of parameters selected based on the observed values. The more inferences are made, the more likely erroneous inferences become. Several statistical techniques have been developed to address that problem, typically by requiring a stricter significance threshold for individual comparisons, so as to compensate for the number of inferences being made. History The problem of multiple comparisons received increased attention in the 1950s with the work of statisticians such as Tukey and Scheffé. Over the ensuing decades, many procedures were developed to address the problem. In 1996, the first international conference on multiple comparison procedures took place in Tel Aviv. Definition Multiple comparisons arise when a statistical analysis involves multiple simultaneous statistical tests, each of wh

Multiple testing with test statistics following heavy-tailed distributions

Zhiwen Jiang

In multiple testing problems where the components come from a mixture model of noise and true effect, we seek to first test for the existence of the non-zero components, and then identify the true alternatives under a fixed significance level

\alpha

. Two parameters, namely the fraction of the non-null components

\varepsilon

and the size of the effects

\mu

, characterise the two-point mixture model under the global alternative. When the number of hypotheses

m

goes to infinity, we are interested in an asymptotic framework where the fraction of the non-null components is vanishing, and the true effects need to be sizable to be detected. Donoho and Jin give an explicit form of the asymptotic detectable boundary based on the Gaussian mixture model under the classic calibration of the parameters of the mixture model. We prove the analogous results for the Cauchy mixture distribution as an example heavy-tailed case. This requires a different formulation of the parameters, which reflects the added difficulties. We also propose a multiple testing procedure based on a filtering approach that can discover the true alternatives. Benjamini and Hochberg (BH) compare the observed

p

-values to a linear threshold curve and reject the null hypotheses from the minimum up to the last up-crossing, and prove the false discovery rate (FDR) is controlled. However, there is an intrinsic difference in heavy-tailed settings. Were we to use the BH procedure we would get a highly variable positive false discovery rate (pFDR). In our study we analyse the distribution of the

p

-values and devise a new multiple testing procedure to combine the usual case and the heavy-tailed case based on the empirical properties of the

p

-values. The filtering approach is designed to eliminate most

p

-values that are more likely to be uniform, while preserving most of the true alternatives. Based on the filtered

p

-values, we estimate the mode

\vartheta

and define the rejection region

\mathscr{R}(\vartheta, \delta)=\left[ \vartheta -\delta/2, \vartheta +\delta/2 \right]

such that the most informative

p

-values are included. The length

\delta

is chosen by controlling the data-dependent estimation of FDR at a desired level.

EPFL2021

Multiple Comparison Procedures for Large Correlated Data with Application to Brain Connectivity Analysis

Djalel Eddine Meskaldji

Recent developments in medical imaging and image analysis allow the determination of interregional brain connectivity through diffusion Magnetic Resonance Imaging (MRI) tractography. The brain connectivity is represented by a connection matrix where each cell represents a certain measure of connectivity between two regions of interest of the brain. An important question related to this challenging field of neuroscience is the ability to learn from connection matrices and to perform rigorous statistical analysis to derive new medical results. In particular, we focus on the level of brain regions of interest. The brain connectivity analysis involves the problem of multiplicity correction or the so-called multiple testing or multiple comparisons problem, which is the principal subject of this thesis. The work described in this thesis can be divided into three essential parts. First, the general problem of multiple comparisons. The objective of this part is to develop new multiple comparisons procedures that have optimal behavior. More precisely, we propose a comprehensive family of error rates together with a corresponding family of multiple comparison procedures. The new family generalizes almost all existing error rates. Second, the problem of multiple comparisons for positively dependent data. By supposing that the brain regions of interest that are in the same vicinity or that belong to the same functional subnetwork are positively correlated, we develop new multiple comparison strategies that exploit this additional information in order to increase sensitivity for detecting real effects. Finally, the application of multiple comparisons to brain connectivity matrices. This part is an application of the two previous parts. In addition, we adapt the statistical methods to brain connectivity analysis by estimating the structure of positive dependence. This third part can be seen as a validation of the statistical methods developed in the first and the second part. The thesis represents a complete framework of an adaptive strategy for the statistical analysis of both functional and structural connection matrices ready to be used in single subject analysis or group comparison. The framework is general enough to be used not only in neuroscience but also in many other research domains. In particular, in the brain connectivity context, it will permit an efficient investigation of a large range of pathologies where changes in brain connectivity are a key ingredient of medical interpretation, such as Alzheimer or Schizophrenia.

EPFL2013