MEPX
Chapter 10 of 13All chapters
  1. 1What analysis is for
  2. 2Types of data
  3. 3Collecting and sampling
  4. 4Cleaning data
  5. 5Describing the middle
  6. 6Describing the spread
  7. 7Choosing a chart
  8. 8Correlation
  9. 9Getting to causation
  10. 10Hypothesis testing
  11. 11Traps and paradoxes
  12. 12Tools of the trade
  13. 13Communicating results

Chapter 10 of 13

Hypothesis testing

What a p-value does and does not say.

The logic

Assume nothing is going on, then ask how surprising your data would be under that assumption. A small p-value means the data would be unusual if nothing were going on.

  • It is not the probability that the conclusion is true.
  • It says nothing about the size or importance of an effect.

Significance and size

With a big enough sample, a meaningless difference becomes statistically significant. Always report the effect size and a confidence interval alongside any test.