Learning rate

How big a step training takes each time. Too big overshoots; too small crawls.

Other terms from the same lesson

• cosine decay: A schedule that lowers the learning rate along a smooth half-wave curve, slowly at first, then faster, then gently down to near zero.
• diverge: When training blows up: the loss grows instead of settling, often because the learning rate is too big.
• learning-rate schedule: A plan for changing the learning rate as training goes on, usually starting larger and ending smaller.
• learning-rate sweep: Training briefly with several learning rates, often each about three times the last, and keeping the one whose loss falls best.
• warmup: Starting training with a tiny learning rate and raising it over the first steps, so early, wild updates don't knock the model off course.