AdamW

Adam with weight decay applied separately from the gradient step. It's the default optimizer for training most language models.

Other terms from the same lesson

• momentum: An optimizer trick: keep a running average of recent steps so training keeps rolling in a steady direction instead of zigzagging.
• optimizer: The rule that decides how to change the weights each step, given the gradients. Adam is the usual choice.