Mixture of experts

Also written: MoE

A model with many specialist sub-networks, where only a few are switched on for each token, saving compute.

Other terms from the same lesson

• active parameters: The parameters each token actually uses in a mixture-of-experts model. They set the compute per token; the total sets the memory needed.
• dense model: An ordinary model where every token uses every parameter, as opposed to a mixture of experts.
• expert: One of the several feed-forward networks in a mixture-of-experts layer. Each token only goes through the few the router picks.
• load-balancing loss: An extra training penalty that pushes the router to spread tokens evenly, so no expert is overloaded and none sits unused.
• router: The small layer in a mixture-of-experts model that scores the experts for each token and sends the token to the top few.
• shared expert: An expert that every token goes through, alongside the ones the router picks.