Momentum

An optimizer trick: keep a running average of recent steps so training keeps rolling in a steady direction instead of zigzagging.

Other terms from the same lesson

• AdamW: Adam with weight decay applied separately from the gradient step. It's the default optimizer for training most language models.
• optimizer: The rule that decides how to change the weights each step, given the gradients. Adam is the usual choice.