MEPX
Chapter 3 of 10All chapters
  1. 1The artificial neuron
  2. 2Layers and depth
  3. 3How it learns
  4. 4Activation functions
  5. 5Architectures
  6. 6Keeping it general
  7. 7Training in practice
  8. 8Transfer learning
  9. 9What they cannot do
  10. 10Learning it yourself

Chapter 3 of 10

How it learns

Loss, gradients and updates.

The loop

Predict, measure the error with a loss function, compute how much each weight contributed to that error, then nudge every weight to reduce it. Repeat for many passes over the data.

  • Backpropagation is the chain rule applied efficiently across the whole network.
  • The learning rate controls step size and is the most sensitive setting.

Batches and epochs

Weights are updated on small batches rather than the whole dataset, which is faster and adds useful noise. One pass over all the data is an epoch.