Residual connection

Also written: residual connections, skip connection, skip connections

A shortcut that adds a layer's input to its output, so information and gradients flow easily through deep networks.

Other terms from the same lesson

• residual block: A layer that adds its result to its own input (x + f(x)), so the input always has a clean path through.
• residual stream: The running total that flows through a transformer: each layer reads it and adds its result back in, instead of replacing it.