What Is Tokenization in NLP? Breaking Text Into Tokens Explained
Tokenization is the step in natural language processing (NLP) that splits a piece of text into smaller units called tokens, which can be words, parts of words, or characters. A tokenizer then maps each token to an identifier from a fixed vocabulary so a model can look up an embedding and work with numbers instead of raw strings. It is the first operation a language model performs on your input, and the way text is split affects how the model handles spelling, punctuation, and unfamiliar words.
Why models need tokenization
A neural network cannot read letters directly. It needs a sequence of identifiers, each pointing to a learned vector. Tokenization provides the bridge between the text you type and those vectors:
- The text is cut into tokens according to a fixed set of rules.
- Each token is matched against a vocabulary, the set of known tokens built during training.
- Every matched token is replaced by an integer ID and then by its embedding.
- The model processes that sequence and converts its output IDs back into text.
Because the vocabulary is fixed, anything not covered by it has to be split further rather than rejected. This is why modern systems rarely split only on spaces.
Word, character, and subword tokens
There are three common granularity levels, each with different trade-offs.
| Granularity | Example for "unhappiness" | Strength | Limitation |
|---|---|---|---|
| Word | unhappiness (one token) |
Easy to read, few tokens | Every new or misspelled word is out of vocabulary |
| Character | u,n,h,a,p… |
Handles any spelling | Very long sequences, weak meaning per token |
| Subword | un + happiness |
Covers new words while keeping meaning | Needs a trained merge or vocabulary rule |
Early systems split on spaces and punctuation, which worked for common words but failed on names, slang, and typos. Character-level models accepted any string but produced long sequences. Most large language models now use subword tokens, splitting an unfamiliar word into pieces the model already knows. For example, "tokenization" can become token + ization, and a rare word can be broken down without a special entry.
Common subword algorithms
Subword vocabularies are built from training data using a few established methods.
Byte-Pair Encoding (BPE)
BPE starts with characters and repeatedly merges the most common adjacent pair until the vocabulary reaches a chosen size. After training, new text is encoded by applying those merges in order. It is widely used because it is simple and effective.
WordPiece
WordPiece is similar but chooses merges based on how much they improve the model's likelihood rather than raw frequency. It is associated with several Google and BERT-family models.
Unigram and SentencePiece
The unigram approach works in the opposite direction: it starts with many candidates and removes the ones that contribute least, so it can give more than one valid segmentation. SentencePiece is a tool that wraps such algorithms and treats text as a raw sequence, which makes it practical for languages without clear spaces between words.
A concrete example
Take the sentence "Chatbots are helpful." A simple word tokenizer might produce:
["Chatbots", "are", "helpful", "."]
A subword tokenizer could keep common words whole while splitting a longer or rarer word:
["Chat", "bots", "are", "helpful", "."]
The punctuation is kept as its own token because it carries meaning, and capitalization is usually preserved so the model can distinguish proper names. The exact split depends on the vocabulary; two models can tokenize the same sentence differently.
Why tokenization is harder than it looks
Splitting text is not uniform across languages and situations.
- Punctuation and contractions: "don't" can be
do+n't, and commas or hyphens need explicit rules. - Morphology: some languages build long words from many parts, so word-level splitting misses structure.
- Writing systems: languages such as Thai and Japanese do not mark word boundaries with spaces, and Vietnamese often combines several syllables into one word.
- Multilingual balance: if a vocabulary is dominated by English, other languages get split into more tokens, which raises cost and can weaken results.
These points explain why tokenization is language-specific work rather than a single universal splitter.
How it relates to other NLP concepts
The tokens produced here are the same AI tokens counted for input length and pricing, and each one is turned into one of the word embeddings the model reads. Older representation methods such as Word2Vec assumed mostly word-level units, while current large language models rely on subword vocabularies to handle open-ended text. The broader field and its tasks are introduced in the NLP overview, and common terms are collected in the machine learning glossary.
FAQ
Is a token the same as a word?
No. A token can be a whole word, a piece of a word, or a single character. In many current models a common word is one token, while longer or rare words are split into several tokens.
Why does tokenization matter for cost?
Models price and limit input by tokens. If your language is split into many small pieces, the same sentence uses more tokens and fills the context window faster than text that stays in longer tokens.
Can I choose the tokenizer myself?
When using a hosted model you use its built-in tokenizer. If you train or fine-tune your own model, you can build a vocabulary suited to your languages and data, which is where BPE or SentencePiece comes in.
Does tokenization understand meaning?
The splitter itself mostly follows statistical rules and does not interpret meaning. Meaning is handled later by embeddings and the model's layers; tokenization only decides the units those layers receive.
Key takeaways
Tokenization converts raw text into the fixed-vocabulary tokens a model can encode as numbers. Subword methods such as BPE and WordPiece balance readable units with the ability to handle unseen words, and choices made here affect sequence length, cost, and multilingual performance. It is a small but foundational step that sits before embeddings and every downstream NLP task.
Sources
- https://web.stanford.edu/class/cs224n/slides_w26/cs224n-2026-guest-julie-tokenization-multilinguality.pdf
- https://web.stanford.edu/~jurafsky/slp3/old_aug25/2.pdf
- https://www.cs.cornell.edu/courses/cs5740/2024sp/slides/06%20-%20tokenization.pdf
- https://www.datacamp.com/blog/what-is-tokenization
- https://www.aiplusinfo.com/blog/what-is-tokenization-in-nlp/
Learn by doing
Read a guide, then practice with bite-sized AI lessons you can take offline.
Start learning free