MEPX
Chapter 3 of 10All chapters
  1. 1Framing the problem
  2. 2Data
  3. 3Training and testing
  4. 4Underfitting and overfitting
  5. 5Measuring performance
  6. 6Common algorithms
  7. 7How training works
  8. 8The pipeline
  9. 9After deployment
  10. 10Common pitfalls

Chapter 3 of 10

Training and testing

Honest evaluation.

The split

Hold back data the model never sees during training, and use it once at the end. A validation set is used for tuning so the test set stays untouched.

  • Cross-validation rotates the held-out part for a more reliable estimate.
  • Time series must be split by time, never randomly.

Leakage

If any information from the future or the test set reaches training, scores become fantastic and useless. It is the most common cause of a model that works in the notebook and fails in production.