Why the same prompt has different token counts
Tokenizers explained without the textbook: vocabulary, chunks, and hidden request overhead.
Updated Jul 25, 2026 · Reviewed by PromptCostLab
Token and pricing information changes frequently. Use this guide for planning and verify model-specific limits and prices with the provider before production billing.
Tokens are learned text pieces
A token is not a fixed number of letters or words. A tokenizer uses a vocabulary of common pieces learned for a model family. A frequent English word might be one token, while an uncommon name can be split into several pieces. Punctuation, whitespace, emoji, source code, and non-English text can behave very differently.
Why providers disagree
Model families can use different vocabularies and preprocessing rules. The same prompt can therefore produce one count for a GPT model and another for Claude, Gemini, Kimi, or an open-weight model. Even models from the same provider may not share an identical tokenizer.
This is why a universal characters-to-tokens formula is an estimate. The common four-characters-per-token shortcut is useful for planning ordinary English text, but it is not an exact tokenizer.
Which counting method should you use?
Use a private local estimate for prompt editing, document sizing, quick comparisons, and context warnings. Use an official tokenizer or counting endpoint for strict limits, billing, tool-heavy agents, and multimodal requests. After the request runs, provider usage data is the authoritative source for the bill.
- Local estimate: fast, private, approximate.
- Tokenizer library: model-specific for supported text formats.
- Provider count endpoint: best before structured requests.
- Usage response: best after the real request.
How to avoid surprises
Keep a safety margin, test the languages and file types your users actually submit, and monitor real usage in production. If exact and estimated counts regularly diverge, update your planning multiplier or route those requests through provider-backed counting.
Prices, token behavior, and model limits change. These primary provider docs are the right place to verify the details before shipping.