What are tokens and the context window in AI?

What a token looks like

Language models do not read letters or whole words. A tokenizer first cuts the text into pieces from a fixed vocabulary, and each piece becomes a number. The model only ever sees these numbers.

The examples below were counted with OpenAI's open-source tokenizer library tiktoken, using the o200k_base encoding. Other models use other tokenizers, so their counts differ, but the pattern is similar.

Common words are a single token, usually including the space before them. Rare or long words are split into pieces. Numbers and dates are split in ways that do not match their meaning, which is one reason models make arithmetic slips. And because strawberry arrives as three chunks, the model never directly sees how many letters r it has.

Why models use subword tokens

A vocabulary of whole words would be enormous and would still miss new words, names and typos. A vocabulary of single characters would make every text very long. Subword tokens are the compromise.

Most tokenizers today use byte pair encoding (BPE) or a close relative. The method starts with single bytes or characters and repeatedly merges the pair that appears most often in the training text into a new token. After many merges, frequent words become single tokens, while rare words can still be built from smaller pieces. Nothing is ever out of vocabulary.

Each model family trains its own tokenizer, so the same sentence gives different counts in different models. Token counts are useful for estimates and budgets, not as a universal unit.

The same sentence in different languages

Tokenizers are trained mostly on the languages that are most common in their training data. Languages with less data, or with different scripts, often need more tokens for the same meaning. Here is one instruction, Please summarize this article in three bullet points, translated and counted with an older tokenizer, cl100k_base, and a newer one, o200k_base:

Two things stand out. The newer tokenizer with a larger vocabulary is much more efficient for Chinese, Japanese, Korean and Vietnamese. And even so, these languages still use more tokens per idea than English in this example. Because API prices and context limits are counted in tokens, the same task can cost more and fill the window faster in some languages.

What the context window is

The context window is the model's working memory for one request. It is separate from what the model learned in training. Everything the model can use to write its answer must fit inside it, and the answer itself counts too.

In a chat app, each new message is sent together with the earlier conversation. So the window fills up as the chat grows. In a typical request, these parts all count:

• The system prompt, the hidden instructions from the app or developer.
• Every earlier user message and assistant reply in the conversation.
• Attached documents, pasted text, images and their extracted text.
• Tool definitions and tool results, such as web search results.
• The new answer, including any hidden reasoning or thinking tokens the model writes before it.

Window sizes vary a lot between models and change often, from a few thousand tokens in small local models to a million or more in some large hosted models at the time of writing. Check the documentation of the model you use.

What happens when the window is full

Different products handle the limit in different ways:

The last row matters most in daily use. Research has shown that models often use information at the start and end of a long input better than information in the middle. Model providers also note that accuracy and recall can drop as the token count grows. A bigger window lets you include more, but it does not guarantee the model will use all of it well.

How to estimate and save tokens

1. Estimate first. For English, multiply words by about 1.3. For Chinese or Japanese, count characters, and expect roughly one token or a bit less per character with newer tokenizers. A word counter, for example cv.cm's word count tool, gives you words and characters quickly.
2. Count exactly when it matters. Use the tokenizer or token-counting endpoint of the model you call, since counts differ between models.
3. Send only what is needed. Paste the relevant section of a document, not the whole file. Remove repeated headers, footers and boilerplate.
4. Put key instructions at the start or end of a long prompt, not buried in the middle.
5. Start a new chat for a new topic. Old messages keep using space and can distract the model.
6. Leave room for the answer. If the input nearly fills the window, the reply will be short or cut off.
7. For long tasks, ask for a summary of decisions so far, then continue in a fresh chat with that summary.

Common mistakes

• Treating tokens as words. A 1,000-word document is not 1,000 tokens, and in many languages it is far more.
• Assuming the model read everything. Even inside the window, details in the middle of a long input can be ignored. Ask the model to quote the passage it relied on.
• Expecting memory across chats. A new conversation starts with an empty window unless the app inserts saved notes.
• Forgetting hidden content. System prompts, tool definitions and reasoning tokens use the window even though you do not see them.
• Comparing limits between models directly. A 100,000-token limit in one model and in another model can hold different amounts of text because their tokenizers differ.