Weight initialization

Also written: initialization, initialisation

Choosing a network's starting weights. Small random numbers of the right size let signals and gradients flow from the first step.

Other terms from the same lesson

• fan-in: How many inputs feed into one neuron. Good starting weights shrink as fan-in grows.
• He initialization: Starting weights sized for ReLU layers: random, with a spread of about √(2 / fan-in), so signals keep a steady size through many layers.
• saturated: When a sigmoid or tanh neuron's input is so large that its output is stuck near the flat end, so its slope (and learning) is almost zero.
• symmetry breaking: Starting weights at different random values so neurons in a layer learn different things instead of staying identical copies.
• Xavier initialization: Starting weights sized for tanh or sigmoid layers, using the average of how many inputs and outputs a layer has.