Tokenization is the process of breaking raw text into tokens — the units a language model actually reads. Different tokenizers handle words, subwords, or characters depending on the model. Word-level tokenization splits on spaces; subword tokenizers like BPE merge common character sequences to handle rare words. Proper tokenization is critical for multilingual models that must handle Arabic, French, and English in the same system.
LLMs & Models
Tokenization
Related terms
Learn to use these concepts in practice.
Join the 212AY Academy →