How tokenization works

From text to token IDs

A tokenizer searches its vocabulary for pieces that represent the input. Frequent patterns are represented compactly as single tokens, while rare strings are split into smaller pieces. The model receives numerical token IDs rather than raw words — the string tokenization might arrive as three IDs like 4463, 866, 445.

Modern tokenizer designs differ in training data, vocabulary size, normalization rules, and whitespace handling. These choices affect efficiency, cost, and the final count — and they change between generations: Anthropic's 2026 tokenizer refresh made the same text split into roughly 30% more tokens than before.

Byte-pair encoding, in plain English

Most current tokenizers are built with byte-pair encoding (BPE) or a close variant. Training starts from raw bytes and repeatedly merges the most frequent adjacent pair into a new vocabulary entry. After millions of merges, the vocabulary contains common words, word pieces, punctuation, and frequent multi-character patterns.

The consequence is frequency-based pricing: text made of common English patterns compresses beautifully (the, and, of the), while rare names, numbers, URLs, and identifiers shatter into many small tokens. That's why casual prose runs ~0.85 words per token and dense technical text closer to 0.65.

Code, emoji, and multilingual text

Source code is the expensive case: indentation, operators, brackets, and identifiers mean code commonly needs 30–50% more tokens than English prose of the same visual length. Emoji can be multiple Unicode code points. Languages written without Latin script — or without spaces at all — can cost several times more tokens per word; CJK text often lands near one token per character.

Test the content your users actually submit instead of applying one English conversion ratio everywhere. A multilingual product should measure each language separately.

Why a browser estimate can differ

A browser tool that runs a real tokenizer — like the PromptCostLab calculator, which loads the o200k vocabulary — counts visible plain text exactly the way an OpenAI model would, and closely for other providers. But a real API payload may also contain role labels, tool schemas, structured messages, images, files, and provider formatting that add tokens the text box never showed.

• Use local tokenizer counting for drafting, comparisons, and sizing.
• Use the provider's counting endpoint for structured payloads before they run.
• Use returned usage fields for final billing truth.
• Expect small differences between families — every provider trains its own vocabulary.

How exact is browser tokenization?

For OpenAI models, the o200k tokenizer is the real thing — counts match what the API sees for plain text. For Claude, Gemini, Grok, Kimi, and DeepSeek, providers do not publish their tokenizers, so tools use o200k as a proxy and label the result an approximation. In practice the agreement is close for ordinary prose and looser for code and non-English text.

When the number must be exact — billing reconciliation, hard production limits — use the provider's counting endpoint (Anthropic's count-tokens, Gemini's countTokens) or read the usage fields after the request runs.

Designing around uncertainty

Keep context headroom of 10–20%, log real usage, and compare estimates with provider results. If your application handles multiple model families, store token and cost assumptions per route rather than pretending one conversion is universal. And re-measure after model upgrades — tokenizer generations change the math silently.