byte-level BPE

BPE that starts from the 256 possible bytes instead of letters, so it can spell out any text at all, emoji included.

Other terms from the same lesson

• BPE: Byte-pair encoding: a way to build a token vocabulary by repeatedly merging the most common pairs of characters.
• pre-tokenization: Splitting text into rough chunks, such as words and spaces, before the tokenizer merges pieces inside each chunk.