MEPX
Chapter 6 of 10All chapters
  1. 1Samples and populations
  2. 2The central limit theorem
  3. 3Confidence intervals
  4. 4Hypothesis testing
  5. 5Multiple comparisons
  6. 6A/B testing
  7. 7Causal inference
  8. 8Regression
  9. 9A Bayesian view
  10. 10Reporting results

Chapter 6 of 10

A/B testing

Experiments in production.

Doing it properly

Randomise assignment, decide the metric and sample size in advance, run for whole business cycles, and analyse once at the end.

  • Peeking and stopping early inflates false positives severely.
  • Weekday and weekend users differ; a partial week biases the result.

Common failures

Users seeing both variants, novelty effects fading after a fortnight, a metric that improves while the business gets worse, and tests too small to detect the effect they were run for.