MEPX
Chapter 6 of 10All chapters
  1. 1The artificial neuron
  2. 2Layers and depth
  3. 3How it learns
  4. 4Activation functions
  5. 5Architectures
  6. 6Keeping it general
  7. 7Training in practice
  8. 8Transfer learning
  9. 9What they cannot do
  10. 10Learning it yourself

Chapter 6 of 10

Keeping it general

Fighting memorisation.

Techniques

Dropout randomly disables units during training. Weight decay penalises large weights. Early stopping halts when validation performance stops improving. Data augmentation invents plausible variations.

  • Each adds a constraint that discourages memorising individual examples.
  • More data remains the strongest regulariser of all.

Batch normalisation

Rescaling activations between layers stabilises training and allows higher learning rates. It became standard because it works, and the full explanation is still debated.