# Tokenizers: Why an LLM Can't Count the R's in Strawberry > Before a language model sees your text, a tokenizer chops it into subword pieces from a fixed vocabulary — and that boundary explains the model's weirdest failures: bad arithmetic, the strawberry problem, why code indentation matters, and why some languages cost 3x more. URL: https://harshith.in/blog/tokenizers-explained/ Author: Harshith Sunku (Staff Engineer) — https://harshith.in/about/ Tags: ai, algorithms, fundamentals The model that can explain quantum field theory cannot reliably count the letters in "strawberry." That contradiction isn't a reasoning failure — it's a *perception* failure, and it happens before the model does any thinking at all, in a preprocessing step most people never look at: the tokenizer. Understanding tokenization explains a surprising share of LLM quirks, and it's pure computer science — a vocabulary, a greedy merge algorithm, and consequences. ## Models Don't See Characters A language model operates on **tokens**, not letters — integer IDs into a fixed vocabulary of maybe 50k–200k entries. The tokenizer maps text to those IDs, and the mapping is *subword*: common words are one token, rare words split into pieces, and the pieces are learned, not designed: ```text "tokenizer" → ["token", "izer"] 2 tokens "strawberry" → ["str", "aw", "berry"] 3 opaque chunks " the" → one token (leading space included!) "1234567" → maybe ["123", "456", "7"] — inconsistent digit grouping ``` The model sees token IDs like [496, 3213], never the characters inside them. So "how many R's in strawberry" asks the model to introspect the *spelling* of chunks it perceives as atomic symbols — roughly like asking you how many times the pixel-shape of a letter appears in a word you only know by sound. It can sometimes fake it from training data about spelling, but the information genuinely isn't in its native representation. Same root cause: reversing strings, character-level edits, and rhyming are all harder than tasks that sound far more complex. ## The Algorithm: Byte-Pair Encoding Most modern tokenizers use **BPE** (byte-pair encoding — originally a 1994 *compression* algorithm, repurposed). Training it is simple and greedy: start with individual bytes/characters, then repeatedly find the most frequent adjacent pair in the corpus and merge it into a new token, for a fixed number of merges. ```text start: l o w e r l o w e s t merge most frequent pair "l o" → "lo" merge "lo w" → "low" ... continue N times result: "low" is one token; rare words stay in small pieces ``` The elegance: frequent sequences (common words, `def `, ``, ` the`) become single tokens automatically, rare ones decompose into reusable parts, and because the base is *bytes*, **nothing is ever unrepresentable** — any Unicode, any binary, any emoji falls back to byte tokens (this is why byte-level BPE beat word-level: no out-of-vocabulary problem, ever). It's a compression codebook that the model then learns to reason over. ## The Consequences That Bite Once you see tokenization as a lossy, frequency-biased compression layer between text and model, a batch of real behaviors explains itself: - **Arithmetic is hard partly because of tokenization.** If "1234" and "1235" tokenize into different chunk groupings, the model can't rely on positional digit structure — the number's *form* is inconsistent. Newer tokenizers deliberately split digits uniformly (one per token, or fixed groups) specifically to help math, a direct fix to a tokenization problem. - **Token cost varies wildly by language.** English is what the vocabulary was optimized on, so it's ~1 token per short word. Many non-Latin scripts and less-represented languages fragment into many more tokens for the same meaning — sometimes 2–3× — which means they cost 2–3× more per API call and consume context budget faster. A real fairness-and-cost issue baked into the vocabulary. - **Context limits are in tokens, not characters.** "8k context" depends on *what* you put in it; code with lots of rare identifiers or a non-English language fills it faster than English prose. - **Code formatting matters more than expected.** Whitespace and indentation tokenize in specific ways (some tokenizers have special handling for runs of spaces); this is part of why models are sensitive to code formatting and why tokenizer improvements for code measurably help. - **The "glitch token" phenomenon.** Strings that appeared in the tokenizer's training corpus but barely in the model's (famously `SolidGoldMagikarp`) got a token ID the model never learned to handle, producing bizarre outputs — a seam between two training stages, visible only if you know the two stages exist. The systems lesson generalizes: there's a lossy encoding layer between the user's data and the model's computation, its design choices leak into behavior, and debugging model quirks often means looking *below* the model at how the input was represented — the same instinct as checking the wire format before blaming the application. ## Takeaways - LLMs perceive subword tokens, not characters — the strawberry/ letter-counting/spelling failures are perception limits, not reasoning ones. - BPE builds the vocabulary by greedily merging frequent byte pairs; byte-level fallback means nothing is ever out-of-vocabulary. - Inconsistent digit tokenization hurts arithmetic — modern tokenizers split digits deliberately to fix it. - Token cost and context consumption vary 2–3× by language/script — a cost and fairness consequence encoded in the vocabulary. - Tokenization is a lossy encoding layer whose choices leak into behavior; debug quirks by inspecting the tokens, not just the prompt.