Some Thoughts on Robustness and Multiple Testing

When testing many null hypotheses, deciding which of them to reject is a subtle game. The prevailing approach consists in deciding on a level and type of control against false rejections (errors of type I) and subject to this constraint to maximize the number of rejections. The two extreme types of control are the FWER (family-wise error rate, probability of at least one false rejection) and FDR (false discovery rate, expected rate of false rejections among all rejections). In this talk, I will discuss two topics. First, how to construct alternatives to these two procedures together with elements of an optimality theory and, second, some considerations about robustness. The FWER and the FDR detect alternatives by comparing the ordered p-values to a boundary; a constant equal to =m for FWER and a boundary that is linearly growing in the rank r equal to r=m for FDR. We will show that if the boundary is equal to s(r)=m, a certain control is implied. By choosing Huber’s clipped linear function s(r) = min(k; r) a family of multiple tests, bridging FWER and FDR is created. How to choose k will also be discussed. The theory of control is based on the fact that the p-values for the true hypotheses are uniformly distributed. If the test is approximate and the nomial p-value is not equal to the real one, this no longer holds. In this case, the control is also merely nominal. We will discuss some examples to illustrate this point and to measure the likely effects on the conclusions of a multiple testing procedure.