MEPX
Chapter 8 of 10All chapters
  1. 1Framing the problem
  2. 2Data
  3. 3Training and testing
  4. 4Underfitting and overfitting
  5. 5Measuring performance
  6. 6Common algorithms
  7. 7How training works
  8. 8The pipeline
  9. 9After deployment
  10. 10Common pitfalls

Chapter 8 of 10

The pipeline

From raw data to a prediction.

Steps

Collect, clean, split, transform, train, evaluate, then deploy. Transformations must be fitted on training data only and applied to the rest, or leakage creeps in.

  • Wrap the steps in a pipeline object so the same order runs everywhere.
  • Version data as well as code; a model is a function of both.

Reproducibility

Fix random seeds, record library versions, and store the exact dataset. Without that, a result cannot be reproduced or debugged later.