Glossary · Phase 5 · Transformers
Language-model head
The last layer of a language model. It turns the final vector at each position into one score per token in the vocabulary.
Learn it properly
The GPT architecture · A stack of decoder blocks that predicts the next token. About 35 minutes.
Other terms from the same lesson
- GPT: Generative pre-trained transformer: a stack of transformer blocks trained to predict the next token.
- weight tying: Reusing the input embedding table as the output layer, so the model needs fewer numbers to learn.