What Is Tokenization in NLP? Breaking Text Into Tokens Explained

Updated · By Viviai

Tokenization is the step in natural language processing (NLP) that splits a piece of text into smaller units called tokens, which can be words, parts of words, or characters. A tokenizer then maps each token to an identifier from a fixed vocabulary so a model can look up an embedding and work with numbers instead of raw strings. It is the first operation a language model performs on your input, and the way text is split affects how the model handles spelling, punctuation, and unfamiliar words.

Why models need tokenization

A neural network cannot read letters directly. It needs a sequence of identifiers, each pointing to a learned vector. Tokenization provides the bridge between the text you type and those vectors:

  1. The text is cut into tokens according to a fixed set of rules.
  2. Each token is matched against a vocabulary, the set of known tokens built during training.
  3. Every matched token is replaced by an integer ID and then by its embedding.
  4. The model processes that sequence and converts its output IDs back into text.

Because the vocabulary is fixed, anything not covered by it has to be split further rather than rejected. This is why modern systems rarely split only on spaces.

Word, character, and subword tokens

There are three common granularity levels, each with different trade-offs.

Granularity Example for "unhappiness" Strength Limitation
Word unhappiness (one token) Easy to read, few tokens Every new or misspelled word is out of vocabulary
Character u,n,h,a,p… Handles any spelling Very long sequences, weak meaning per token
Subword un + happiness Covers new words while keeping meaning Needs a trained merge or vocabulary rule

Early systems split on spaces and punctuation, which worked for common words but failed on names, slang, and typos. Character-level models accepted any string but produced long sequences. Most large language models now use subword tokens, splitting an unfamiliar word into pieces the model already knows. For example, "tokenization" can become token + ization, and a rare word can be broken down without a special entry.

Common subword algorithms

Subword vocabularies are built from training data using a few established methods.

Byte-Pair Encoding (BPE)

BPE starts with characters and repeatedly merges the most common adjacent pair until the vocabulary reaches a chosen size. After training, new text is encoded by applying those merges in order. It is widely used because it is simple and effective.

WordPiece

WordPiece is similar but chooses merges based on how much they improve the model's likelihood rather than raw frequency. It is associated with several Google and BERT-family models.

Unigram and SentencePiece

The unigram approach works in the opposite direction: it starts with many candidates and removes the ones that contribute least, so it can give more than one valid segmentation. SentencePiece is a tool that wraps such algorithms and treats text as a raw sequence, which makes it practical for languages without clear spaces between words.

A concrete example

Take the sentence "Chatbots are helpful." A simple word tokenizer might produce:

["Chatbots", "are", "helpful", "."]

A subword tokenizer could keep common words whole while splitting a longer or rarer word:

["Chat", "bots", "are", "helpful", "."]

The punctuation is kept as its own token because it carries meaning, and capitalization is usually preserved so the model can distinguish proper names. The exact split depends on the vocabulary; two models can tokenize the same sentence differently.

Why tokenization is harder than it looks

Splitting text is not uniform across languages and situations.

These points explain why tokenization is language-specific work rather than a single universal splitter.

How it relates to other NLP concepts

The tokens produced here are the same AI tokens counted for input length and pricing, and each one is turned into one of the word embeddings the model reads. Older representation methods such as Word2Vec assumed mostly word-level units, while current large language models rely on subword vocabularies to handle open-ended text. The broader field and its tasks are introduced in the NLP overview, and common terms are collected in the machine learning glossary.

FAQ

Is a token the same as a word?

No. A token can be a whole word, a piece of a word, or a single character. In many current models a common word is one token, while longer or rare words are split into several tokens.

Why does tokenization matter for cost?

Models price and limit input by tokens. If your language is split into many small pieces, the same sentence uses more tokens and fills the context window faster than text that stays in longer tokens.

Can I choose the tokenizer myself?

When using a hosted model you use its built-in tokenizer. If you train or fine-tune your own model, you can build a vocabulary suited to your languages and data, which is where BPE or SentencePiece comes in.

Does tokenization understand meaning?

The splitter itself mostly follows statistical rules and does not interpret meaning. Meaning is handled later by embeddings and the model's layers; tokenization only decides the units those layers receive.

Key takeaways

Tokenization converts raw text into the fixed-vocabulary tokens a model can encode as numbers. Subword methods such as BPE and WordPiece balance readable units with the ability to handle unseen words, and choices made here affect sequence length, cost, and multilingual performance. It is a small but foundational step that sits before embeddings and every downstream NLP task.

Sources

  1. https://web.stanford.edu/class/cs224n/slides_w26/cs224n-2026-guest-julie-tokenization-multilinguality.pdf
  2. https://web.stanford.edu/~jurafsky/slp3/old_aug25/2.pdf
  3. https://www.cs.cornell.edu/courses/cs5740/2024sp/slides/06%20-%20tokenization.pdf
  4. https://www.datacamp.com/blog/what-is-tokenization
  5. https://www.aiplusinfo.com/blog/what-is-tokenization-in-nlp/

Learn by doing

Read a guide, then practice with bite-sized AI lessons you can take offline.

Start learning free