MEPX
Chapter 5 of 10All chapters
  1. 1Samples and populations
  2. 2The central limit theorem
  3. 3Confidence intervals
  4. 4Hypothesis testing
  5. 5Multiple comparisons
  6. 6A/B testing
  7. 7Causal inference
  8. 8Regression
  9. 9A Bayesian view
  10. 10Reporting results

Chapter 5 of 10

Multiple comparisons

Testing until something works.

The problem

Every test carries a false positive chance. Run twenty at the 5 percent level and expect one significant result from pure noise.

  • Corrections such as Bonferroni or false discovery rate control this.
  • Dashboards with dozens of metrics are a multiple comparisons machine.

Researcher degrees of freedom

Choosing the outcome, the subgroup or the exclusion rule after seeing the data inflates false positives enormously. Pre-registering the plan is the fix.