MEPX
Chapter 11 of 13All chapters
  1. 1What analysis is for
  2. 2Types of data
  3. 3Collecting and sampling
  4. 4Cleaning data
  5. 5Describing the middle
  6. 6Describing the spread
  7. 7Choosing a chart
  8. 8Correlation
  9. 9Getting to causation
  10. 10Hypothesis testing
  11. 11Traps and paradoxes
  12. 12Tools of the trade
  13. 13Communicating results

Chapter 11 of 13

Traps and paradoxes

Where careful people still get caught.

Simpson's paradox

A trend can appear in every subgroup and reverse when the groups are combined, usually because the groups differ in size and base rate. Always check whether a total hides opposing patterns.

  • Testing many hypotheses guarantees some will look significant by chance.
  • Deciding what to test after seeing the data invalidates the test.

Regression to the mean

Extreme results tend to be followed by less extreme ones with no cause at all. Much of the apparent effect of an intervention on the worst performers is this and nothing more.