Weight decay

Shrinking every weight a tiny bit on each step, so the model keeps its weights small and is less likely to overfit.

Other terms from the same lesson

• elastic net: Regularization that mixes the ridge and lasso penalties.
• lasso: Regularization that adds the plain size of every weight to the loss. It pushes unhelpful weights to exactly zero, so it also picks features.
• regularization: Gently penalising a model for being too complicated, to stop it overfitting.
• ridge regression: Regularization that adds the squared size of every weight to the loss, so weights stay small but rarely reach exactly zero.