What are AI tokens?

Tokens are not the same as words

A tokenizer breaks text into vocabulary pieces. A common word may be one token, while an unusual name, emoji, code symbol, or word in another language may use several. Spaces and punctuation can also affect the result. That is why word count and token count are related but never interchangeable.

For ordinary English prose, rough character-based estimates are useful for planning. They should not be presented as exact counts for every model or language.

Where tokens appear in an AI request

Input tokens include the material sent to the model: system instructions, user messages, conversation history, retrieved documents, and sometimes tool definitions or file content. Output tokens are generated by the model. Providers commonly price these categories separately.

• Input: instructions, questions, context, and history.
• Output: the model’s answer.
• Cached input: reused prompt prefixes when supported.
• Hidden overhead: request formatting and structured schemas.

Why token count matters

Tokens determine whether a request fits in a model’s context window and contribute to API cost. A short user message can still become a large request if the application attaches a long system prompt, many documents, or an entire conversation.

Measure the assembled production request, reserve space for output, and keep a safety margin. Use provider usage data when an exact bill matters.

A useful planning workflow

Count the stable prompt first, then add typical conversation and retrieval context. Set the expected response length and multiply the estimated request cost by realistic traffic. Repeat the calculation with a worst-case request so a large customer document or long conversation does not surprise the system.