Pre-tokenization

Splitting text into rough chunks, such as words and spaces, before the tokenizer merges pieces inside each chunk.

Other terms from the same lesson

• BPE: Byte-pair encoding: a way to build a token vocabulary by repeatedly merging the most common pairs of characters.
• byte-level BPE: BPE that starts from the 256 possible bytes instead of letters, so it can spell out any text at all, emoji included.