AI Tokenization Explainer
Tokenization is the step that converts raw input into units a model can process. For text, a tokenizer may represent a whole word, part of a word, punctuation mark, space pattern, or individual character as a token. Each token is mapped to a numeric identifier before it enters the model.
Modern language systems often use subword tokenization. Common pieces can remain intact while rare words are split into smaller reusable parts. This gives the model a manageable vocabulary and a way to represent new names or technical terms without reserving a separate entry for every possible word.
Token boundaries are model-specific. The same sentence can use different token counts in different systems, and a token is not the same thing as a word. Spelling, language, whitespace, code, and punctuation can all change the count. Images may also be divided into patches that function as input tokens.
Tokenization matters because context limits, processing cost, and model attention are usually measured in tokens. A poor fit between a tokenizer and a language or domain can make sequences longer and learning less efficient. Developers therefore examine coverage, compression, and how consistently meaningful patterns are represented.
Tokenization converts text or other input into numbered units that a model can process.
Key terms include token, tokenizer, vocabulary, subword, character, token ID, encoding, context length and input sequence.
The tokenizer applies a fixed vocabulary and segmentation rules, then replaces each selected piece with its numeric ID. Special tokens may mark boundaries or instructions.
The model turns those IDs into vectors and processes their relationships; decoding reverses the mapping when output tokens are converted back into readable text.
A familiar word may be one token in one vocabulary and several in another. Less represented languages and unusual strings can require more pieces.
Token counts affect how much material fits in a context window, but they do not directly measure the difficulty or meaning of a passage.
Explore more "Explainers"
Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.