MEPX
Chapter 4 of 10All chapters
  1. 1Samples and populations
  2. 2The central limit theorem
  3. 3Confidence intervals
  4. 4Hypothesis testing
  5. 5Multiple comparisons
  6. 6A/B testing
  7. 7Causal inference
  8. 8Regression
  9. 9A Bayesian view
  10. 10Reporting results

Chapter 4 of 10

Hypothesis testing

The machinery and its traps.

The logic

Assume no effect, compute how surprising the data would be under that assumption, and report it as a p-value. A small value means the data is unusual if nothing is happening.

  • It is not the probability that the null hypothesis is true.
  • Failing to reject is not evidence of no effect, especially with small samples.

Power

Power is the chance of detecting a real effect of a given size. Low power both misses effects and exaggerates the ones it finds, which drives much of the replication problem.