Weight tying

Also written: tied weights

Reusing the input embedding table as the output layer, so the model needs fewer numbers to learn.

Other terms from the same lesson

• GPT: Generative pre-trained transformer: a stack of transformer blocks trained to predict the next token.
• language-model head: The last layer of a language model. It turns the final vector at each position into one score per token in the vocabulary.