Mixture of experts

Why it matters

Many of the largest open models (DeepSeek-V3, Mixtral, Llama 4, Qwen's biggest models) are mixtures of experts, and several frontier labs are widely believed to use the technique too. MoE is how a model can have hundreds of billions of parameters but run each token through only a small fraction of them. It explains spec sheets that list two parameter counts, and why some models are cheap per token but still need a lot of GPU memory.

What you'll be able to do

• Explain how a mixture-of-experts layer routes each token to a few expert MLPs
• Distinguish total parameters from active parameters, and what each costs
• Train a small MoE layer and see why routing needs a load-balancing loss
• Read MoE model specs such as '671B total, 37B active'

Before this lesson

• The transformer block

Next in this phase: Multimodal transformers