Tokenization

Also written: tokenizer, tokenizers, tokenize

Splitting text into tokens, and turning each into a number the model can use.

Other terms from the same lesson

• context window: The most text a model can take in at once, measured in tokens: its short-term memory.
• proxy: A server that passes your requests on to another service, often adding the secret key so it never reaches the browser.
• token: A chunk of text a model reads and writes, often a word or part of a word. Models count and charge by tokens.
• token ID: The number a tokenizer gives each token. Models work with these numbers, not with the text itself.