Dropout

Why it matters

Large networks can memorize small datasets. Dropout is the simplest strong defense: during training, randomly switch off a fraction of the neurons on every step, so the network can't rely on any single one. It takes one line of code, and you'll find it in the original transformer and in many fine-tuning recipes, often with the rate as one of the few knobs exposed.

What you'll be able to do

• Explain dropout: randomly zeroing activations during training, and scaling the rest
• Show why inverted dropout keeps the expected activation unchanged
• Measure dropout's effect on an overfitting network
• Place dropout correctly and switch it off for evaluation

Before this lesson

• The training loop
• Regularization

Next in this phase: Vanishing and exploding gradients