Phase 4 · Deep learning · Architectures
Recurrent networks
Process sequences step by step, and their limits.
Open this lesson Create a free account
Why it matters
Language is a sequence, and before transformers, recurrent networks were how neural networks read and wrote it: machine translation, speech recognition and early text generation all ran on them. Training one yourself shows the core idea every language model shares (predict the next token from what came before) and the two weaknesses, forgetting and sequential processing, that the transformer was designed to fix.
What you'll be able to do
- Explain how a recurrent network carries a hidden state through a sequence
- Train a character-level RNN language model and sample text from it
- Show why gradients vanish across long sequences, and what LSTMs changed
- Explain the limits that led to transformers
Before this lesson
Next in this phase: Embedding layers