Glossary · Phase 5 · Transformers
Pretraining
The first, huge training run where a model learns language by predicting the next token across vast amounts of text.
Learn it properly
Pretraining at scale · Data, compute and what models learn from the internet. About 35 minutes.
Other terms from the same lesson
- base model: A model straight out of pretraining: it continues text but doesn't yet follow instructions.
- Jaccard similarity: How alike two sets are: the number of items they share, divided by the number of different items in both. 1 means identical, 0 means nothing in common.
- MinHash: A hashing trick that estimates Jaccard similarity from short signatures, so near-duplicate documents can be found without comparing every pair.
- shingle: A short run of words in a row, such as three, taken from a document. Comparing two documents' shingles shows how much text they share.
- utilization: The share of a chip's peak speed that a training run actually gets. Waiting on memory and on other chips keeps it well below 100%.