SwiGLU

A feed-forward layer design used in Llama-style models, where one set of numbers acts as a gate that turns another set up or down.

Other terms from the same lesson

• decoder: A transformer that reads left to right and predicts the next token; each token can only see the ones before it. GPT-style chat models are decoders.
• feed-forward network: The small neural network inside each transformer block. It works on each token on its own, after attention has mixed in information from the other tokens.
• model card: The fact sheet published with a model: its size, what it was trained on, what it's for and its known limits.
• transformer: The design behind today's language models: layers of attention plus small neural networks, repeated many times.
• transformer block: One repeating unit of a transformer: attention, then a small neural network, each with a shortcut and normalization.