What Is a Token in AI? A Beginner-Friendly Explanation

Updated · By Viviai

A token is a small chunk of text that an AI model reads and writes — often part of a word rather than a whole word. Tokens matter because they decide how much text a model can handle at once and how much using an AI API costs.

If you have seen pricing pages listing "per 1 million tokens," this guide explains exactly what that means.

The short definition

A token is the basic unit an LLM uses to process language. A word can be one token, but longer or less common words are often split into several pieces. Punctuation and spaces can also count as tokens.

As a rough rule of thumb for English, 100 tokens equal about 75 words. The exact number varies by language and word choice.

Why models do not just use words

Splitting text into tokens is efficient. Consider the words unhelpful, unhappy, and unusual. They share the prefix un-. Tokenization lets the model recognize that shared piece instead of treating every word as a brand-new object.

It also helps the model handle:

Tokens in practice: how text gets split

Here is the idea, not exact output for any specific model:

The model never sees letters or words directly — it sees tokens.

Tokens and the context window

The context window is the maximum number of tokens a model can consider in a single request, including:

If the context window is 200,000 tokens, the combined input and output must fit within that limit. Once a conversation grows beyond the window, the oldest content drops out and the model can no longer reference it.

This is why pasting a very long document sometimes produces an error or a model that "forgets" the beginning.

Tokens and API pricing

When developers pay to use an AI through an API, pricing is based on tokens:

Output tokens usually cost more than input tokens because generation requires more computation. A price listed as "$2 per million input tokens" means you pay according to the volume of text processed, not per question or per minute.

Understanding tokens helps explain why a short question costs almost nothing while feeding in a 50-page report adds up.

Tokens differ across languages

The same sentence can use different token counts in different languages. This has practical consequences:

This is one reason multilingual projects track token efficiency rather than word counts.

Frequently asked questions

Is a token the same as a word? No. A token can be a whole word, part of a word, punctuation, or a space. On average, expect fewer words than tokens.

Do I count tokens when using a free chat app? You usually do not see them, but the limits still exist behind the scenes. Token counts become visible mainly when using developer APIs.

What happens if I exceed the context window? The model cannot process everything at once. You may get an error, or older content may be cut off. Solutions include shortening the input or processing it in chunks.

Why do output tokens cost more? Generating each new token requires computation, while input tokens are processed once. The pricing reflects that difference.

Key takeaways

Read next: what AI hallucination is and how the context window works in real use.

Sources

  1. https://learn.microsoft.com/es-mx/azure/cloud-adoption-framework/ai-agents/

Learn by doing

Read a guide, then practice with bite-sized AI lessons you can take offline.

Start learning free