Phase 4 · Deep learning · Neural networks
Build an autograd engine
Write backprop yourself in ~100 lines (micrograd).
Open this lesson Create a free account
Why it matters
Writing backward passes by hand, as in the last lesson, works but doesn't scale: every new layer means new derivations and new bugs. Automatic differentiation records operations as they happen and does the backward pass for you. PyTorch's autograd is exactly this idea, applied to tensors. Building a small version yourself (Andrej Karpathy's "micrograd" is the famous one) takes about 100 lines and removes the last bit of mystery from how networks learn.
What you'll be able to do
- Build a Value class that records how each number was computed
- Give each operation a small backward function that applies its local derivative
- Run backward() over the recorded graph in reverse topological order
- Train a tiny neural network using nothing but your own engine
Before this lesson
Next in this phase: Weight initialization