How tokenization works

From text to token IDs

A tokenizer searches its vocabulary for text pieces that represent the input. Frequent patterns can be represented compactly, while rare strings are split into smaller pieces. The model receives numerical token IDs rather than raw words.

Modern tokenizer designs differ in their training data, vocabulary size, normalization rules, and handling of whitespace. These choices affect both efficiency and the final count.

Code, emoji, and multilingual text

Source code contains punctuation, indentation, identifiers, and uncommon strings. Emoji may include multiple Unicode code points. Languages without English-like spacing can split differently again. Test the content your users actually submit instead of applying one English conversion ratio everywhere.

Why a browser estimate can differ

A browser estimator can calculate a fast approximation from visible text. A real API payload may also contain role labels, tool schemas, structured messages, images, files, and provider formatting. Those additions are why PromptCostLab labels local counts as estimates.

• Use local counting for drafting and comparisons.
• Use a model tokenizer for supported plain text.
• Use a provider endpoint for structured payloads.
• Use returned usage for final billing.

Designing around uncertainty

Keep context headroom, log real usage, and compare estimates with provider results. If your application handles multiple model families, store token and cost assumptions per route rather than pretending one conversion is universal.