What Is a Token in AI? A Beginner-Friendly Explanation
A token is a small chunk of text that an AI model reads and writes — often part of a word rather than a whole word. Tokens matter because they decide how much text a model can handle at once and how much using an AI API costs.
If you have seen pricing pages listing "per 1 million tokens," this guide explains exactly what that means.
The short definition
A token is the basic unit an LLM uses to process language. A word can be one token, but longer or less common words are often split into several pieces. Punctuation and spaces can also count as tokens.
As a rough rule of thumb for English, 100 tokens equal about 75 words. The exact number varies by language and word choice.
Why models do not just use words
Splitting text into tokens is efficient. Consider the words unhelpful, unhappy, and unusual. They share the prefix un-. Tokenization lets the model recognize that shared piece instead of treating every word as a brand-new object.
It also helps the model handle:
- Long or compound words.
- Words it has never seen in full.
- Numbers, punctuation, and code.
- Languages with different writing systems.
Tokens in practice: how text gets split
Here is the idea, not exact output for any specific model:
- Short common words like "the" or "and" are often one token.
- A longer word like "understanding" might become two or more tokens.
- A sentence is broken into a sequence of tokens the model reads in order.
- When the model replies, it generates new tokens one at a time.
The model never sees letters or words directly — it sees tokens.
Tokens and the context window
The context window is the maximum number of tokens a model can consider in a single request, including:
- Your prompt.
- Any past conversation.
- Documents you paste in.
- The model's own reply.
If the context window is 200,000 tokens, the combined input and output must fit within that limit. Once a conversation grows beyond the window, the oldest content drops out and the model can no longer reference it.
This is why pasting a very long document sometimes produces an error or a model that "forgets" the beginning.
Tokens and API pricing
When developers pay to use an AI through an API, pricing is based on tokens:
- Input tokens — text you send, including prompts and context.
- Output tokens — text the model generates.
Output tokens usually cost more than input tokens because generation requires more computation. A price listed as "$2 per million input tokens" means you pay according to the volume of text processed, not per question or per minute.
Understanding tokens helps explain why a short question costs almost nothing while feeding in a 50-page report adds up.
Tokens differ across languages
The same sentence can use different token counts in different languages. This has practical consequences:
- Some languages use more tokens per idea, raising API costs.
- A context window that fits 150,000 words of English may fit fewer words in another language.
- Translating a long document can change how much fits in the window.
This is one reason multilingual projects track token efficiency rather than word counts.
Frequently asked questions
Is a token the same as a word? No. A token can be a whole word, part of a word, punctuation, or a space. On average, expect fewer words than tokens.
Do I count tokens when using a free chat app? You usually do not see them, but the limits still exist behind the scenes. Token counts become visible mainly when using developer APIs.
What happens if I exceed the context window? The model cannot process everything at once. You may get an error, or older content may be cut off. Solutions include shortening the input or processing it in chunks.
Why do output tokens cost more? Generating each new token requires computation, while input tokens are processed once. The pricing reflects that difference.
Key takeaways
- A token is the smallest text unit an AI processes, often part of a word.
- Roughly 100 tokens correspond to about 75 English words.
- Context windows and API pricing are both measured in tokens.
- Token counts vary by language, affecting both limits and cost.
Read next: what AI hallucination is and how the context window works in real use.
Sources
Learn by doing
Read a guide, then practice with bite-sized AI lessons you can take offline.
Start learning free