MEPX
Chapter 7 of 10All chapters
  1. 1The artificial neuron
  2. 2Layers and depth
  3. 3How it learns
  4. 4Activation functions
  5. 5Architectures
  6. 6Keeping it general
  7. 7Training in practice
  8. 8Transfer learning
  9. 9What they cannot do
  10. 10Learning it yourself

Chapter 7 of 10

Training in practice

Why it is fiddly.

What goes wrong

Loss not decreasing, exploding to infinity, or falling on training while rising on validation. Each has a short list of usual causes, starting with the learning rate.

  • Always overfit a tiny subset first: if it cannot, something is broken.
  • Initialisation and normalisation of inputs matter more than beginners expect.

Compute

Training large models needs specialised hardware because the operations are enormous matrix multiplications. That is exactly what graphics processors were built for.