Glossary · Phase 4 · Deep learning
Fan-in
How many inputs feed into one neuron. Good starting weights shrink as fan-in grows.
Learn it properly
Weight initialization · Start weights at the right scale. About 25 minutes.
Other terms from the same lesson
- He initialization: Starting weights sized for ReLU layers: random, with a spread of about √(2 / fan-in), so signals keep a steady size through many layers.
- saturated: When a sigmoid or tanh neuron's input is so large that its output is stuck near the flat end, so its slope (and learning) is almost zero.
- symmetry breaking: Starting weights at different random values so neurons in a layer learn different things instead of staying identical copies.
- weight initialization: Choosing a network's starting weights. Small random numbers of the right size let signals and gradients flow from the first step.
- Xavier initialization: Starting weights sized for tanh or sigmoid layers, using the average of how many inputs and outputs a layer has.