Testing hypotheses suggested by the data

In statistics, hypotheses suggested by a given dataset, when tested with the same dataset that suggested them, are likely to be accepted even when they are not true. This is because circular reasoning (double dipping) would be involved: something seems true in the limited data set; therefore we hypothesize that it is true in general; therefore we wrongly test it on the same, limited data set, which seems to confirm that it is true. Generating hypotheses based on data already observed, in the absence of testing them on new data, is referred to as post hoc theorizing (from Latin post hoc, "after this"). The correct procedure is to test any hypothesis on a data set that was not used to generate the hypothesis. Testing a hypothesis suggested by the data can very easily result in false positives (type I errors). If one looks long enough and in enough different places, eventually data can be found to support any hypothesis. Yet, these positive data do not by themselves constitute evidence that the hypothesis is correct. The negative test data that were thrown out are just as important, because they give one an idea of how common the positive results are compared to chance. Running an experiment, seeing a pattern in the data, proposing a hypothesis from that pattern, then using the same experimental data as evidence for the new hypothesis is extremely suspect, because data from all other experiments, completed or potential, has essentially been "thrown out" by choosing to look only at the experiments that suggested the new hypothesis in the first place. A large set of tests as described above greatly inflates the probability of type I error as all but the data most favorable to the hypothesis is discarded. This is a risk, not only in hypothesis testing but in all statistical inference as it is often problematic to accurately describe the process that has been followed in searching and discarding data. In other words, one wants to keep all data (regardless of whether they tend to support or refute the hypothesis) from "good tests", but it is sometimes difficult to figure out what a "good test" is.

À propos de ce résultat

Cette page est générée automatiquement et peut contenir des informations qui ne sont pas correctes, complètes, à jour ou pertinentes par rapport à votre recherche. Il en va de même pour toutes les autres pages de ce site. Veillez à vérifier les informations auprès des sources officielles de l'EPFL.

Concepts associés (4)

Cours associés (8)

Séances de cours associées (59)

Testing hypotheses suggested by the data

Multiple comparisons problem

In statistics, the multiple comparisons, multiplicity or multiple testing problem occurs when one considers a set of statistical inferences simultaneously or infers a subset of parameters selected based on the observed values. The more inferences are made, the more likely erroneous inferences become. Several statistical techniques have been developed to address that problem, typically by requiring a stricter significance threshold for individual comparisons, so as to compensate for the number of inferences being made.

Data dredging

vignette|Exemple de Data dredging. Le data dredging (littéralement le dragage de données mais mieux traduit comme étant du triturage de données) est une technique statistique qui . Une des formes du data dredging est de partir de données ayant un grand nombre de variables et un grand nombre de résultats, et de choisir les associations qui sont « statistiquement significatives », au sens de la valeur p (on parle aussi de p-hacking).

MATH-562: Statistical inference

Inference from the particular to the general based on probability models is central to the statistical method. This course gives a graduate-level account of the main ideas of statistical inference.

MATH-234(d): Probability and statistics

Ce cours enseigne les notions élémentaires de la théorie de probabilité et de la statistique, tels que l'inférence, les tests et la régression.

MATH-234(b): Probability and statistics

Ce cours enseigne les notions élémentaires de la théorie de probabilité et de la statistique, tels que l'inférence, les tests et la régression.

Conception expérimentale en biostatistique MATH-493: Applied biostatistics

Introduit la conception expérimentale en biostatistiques, couvrant le processus de recherche, les tests d'hypothèses, la modélisation ANOVA et l'interprétation des résultats.

Analyse des écarts : principes et applications MATH-236: Probability and statistics II

Explore les principes et les applications de l'analyse des écarts (ANOVA), y compris les hypothèses de test, les modèles, les hypothèses et les tests post-hoc.

Comprendre les statistiques et le design expérimental BIO-449: Understanding statistics and experimental design

Couvre la théorie des probabilités de base, la théorie de la détection des signaux, les statistiques et les méta-statistiques, expliquant la taille des effets, la puissance et les tests d'hypothèses.