Saturated

Also written: saturation, saturate

When a sigmoid or tanh neuron's input is so large that its output is stuck near the flat end, so its slope (and learning) is almost zero.

Other terms from the same lesson

• fan-in: How many inputs feed into one neuron. Good starting weights shrink as fan-in grows.
• He initialization: Starting weights sized for ReLU layers: random, with a spread of about √(2 / fan-in), so signals keep a steady size through many layers.
• symmetry breaking: Starting weights at different random values so neurons in a layer learn different things instead of staying identical copies.
• weight initialization: Choosing a network's starting weights. Small random numbers of the right size let signals and gradients flow from the first step.
• Xavier initialization: Starting weights sized for tanh or sigmoid layers, using the average of how many inputs and outputs a layer has.