MEPX
Chapter 5 of 10All chapters
  1. 1The artificial neuron
  2. 2Layers and depth
  3. 3How it learns
  4. 4Activation functions
  5. 5Architectures
  6. 6Keeping it general
  7. 7Training in practice
  8. 8Transfer learning
  9. 9What they cannot do
  10. 10Learning it yourself

Chapter 5 of 10

Architectures

Shapes for different data.

Three families

Convolutional networks share small filters across an image, which suits spatial data. Recurrent networks process sequences step by step. Transformers use attention to relate all positions at once.

  • Convolution encodes the assumption that a pattern means the same thing anywhere.
  • Transformers replaced recurrent models for language because they parallelise.

Attention

Attention lets each position weigh how much every other position matters. That is what allows a model to connect a pronoun with a noun forty words earlier.