MEPX
Chapter 5 of 10All chapters
  1. 1Framing the problem
  2. 2Data
  3. 3Training and testing
  4. 4Underfitting and overfitting
  5. 5Measuring performance
  6. 6Common algorithms
  7. 7How training works
  8. 8The pipeline
  9. 9After deployment
  10. 10Common pitfalls

Chapter 5 of 10

Measuring performance

Choosing the right number.

Beyond accuracy

With rare events, accuracy is misleading. Precision asks how many predicted positives were right; recall asks how many real positives were found. They trade against each other.

  • A confusion matrix shows all four outcomes and is worth reading directly.
  • For regression, mean absolute error is easier to interpret than squared error.

Match the metric to the cost

Missing a fraud and blocking a legitimate customer have different costs. The metric should reflect that asymmetry rather than treating errors as equal.