Models don't see letters or words — they see tokens, integer IDs for chunks of text. Tokenization is the step that splits text into those chunks, and it quietly shapes cost, context limits, and quirks.
Tokens aren't words
Most LLMs use subword tokenization (e.g. Byte-Pair Encoding) — common words are one token, rare words split into pieces:
"tokenization" → ["token", "ization"]
"cat" → ["cat"]
"🤖" → several bytes/tokensRule of thumb: ~4 characters ≈ 1 token, or ~0.75 words.
Why it matters
| Consequence | Because |
|---|---|
| Cost | APIs bill per token; context is measured in tokens |
| The "strawberry" problem | Models can't easily count letters they never see as letters |
| Non-English costs more | Languages with less training data fragment into more tokens |
| Weird edge cases | Odd spacing/Unicode tokenizes unexpectedly |
Your context window and your bill are both counted in tokens, not words — see Ollama Parameters Guide for
num_ctx.
Related: What Are Embeddings · Context Windows Explained · How Transformers Work