MEPX
Chapter 4 of 10All chapters
  1. 1The artificial neuron
  2. 2Layers and depth
  3. 3How it learns
  4. 4Activation functions
  5. 5Architectures
  6. 6Keeping it general
  7. 7Training in practice
  8. 8Transfer learning
  9. 9What they cannot do
  10. 10Learning it yourself

Chapter 4 of 10

Activation functions

Small choices with large effects.

The common ones

ReLU passes positive values and zeroes negatives, and is the default for hidden layers. Sigmoid and softmax turn outputs into probabilities at the end.

  • ReLU largely solved the vanishing gradient problem that stalled deep networks.
  • Softmax is for choosing one class among several.

Why it mattered

Deep networks were known long before they worked. Better activations, initialisation and hardware are what made training them practical.