A model doesn't read text: what a tokenizer decides for you

A lexer with a different goal

If you’ve written a compiler, you’ve written a tokenizer. The lexer takes characters and emits a stream of symbols: identifiers, keywords, literals. Its goal is to recognise structure so the parser can build a tree.

A model’s tokenizer has the same shape and a different goal. There’s no structure to recognise. The goal is to compress: turn a string into as few ids as possible from a fixed vocabulary, so the model sees fewer positions and the vocabulary stays small enough to put a row of the embedding matrix behind every id.

Both extremes fail. Characters give a tiny vocabulary and sequences that are far too long; every position costs attention, and attention costs the square of the length. Whole words give short sequences and an unbounded vocabulary, plus a hole for every word that isn’t in it. The data scientist knows this hole as out-of-vocabulary; the software engineer knows it as an unhandled case.

Subwords are the compromise. Sennrich, Haddow and Birch (2016) took byte-pair encoding, a compression algorithm from 1994, and pointed it at this problem instead: learn the sub-word units from the corpus rather than declaring them up front.

BPE is one loop

I read tiktoken’s _educational.py. Its encoder is a readable stand-in for the Rust core, which I checked by running both over every sample below and comparing ids. Its trainer is a stand-in for nothing: the Rust core ships no trainer, so bpe_train (_educational.py:119-185) is the only version of the loop in the package. It is one loop. The body of bpe_train:

while len(ranks) < vocab_size:
    stats = collections.Counter()
    for piece in words:
        for pair in zip(piece[:-1], piece[1:]):
            stats[pair] += 1
    most_common_pair = max(stats, key=lambda x: stats[x])
    token_bytes = most_common_pair[0] + most_common_pair[1]
    token = len(ranks)
    # Add the new token!
    ranks[token_bytes] = token
    # ... then replace that pair through the corpus, and loop

Start with 256 tokens, one per byte. Count every adjacent pair across the corpus. Take the most frequent pair, give it the next id, replace it through the corpus left to right. Repeat until the vocabulary is the size you asked for.

The vocabulary is the list of merges, in order. That order is the whole model of the language. Encoding replays it: split the input into bytes, find the adjacent pair with the lowest merge rank, merge it, repeat until nothing merges. bpe_encode (_educational.py:83-116) is thirty-four lines. It never iterates the merge list; training takes the most frequent pair and encoding takes the lowest-ranked one, which comes to the same thing.

The widget runs that loop on a toy corpus, one merge per click. Twelve of its eighteen steps are ties, so the tie-break rule (first pair seen wins) is doing real work.

Two details matter for the engineer. First, it works on bytes, not characters, so any input encodes and there’s no unknown token. A character the merges never saw costs up to one token per UTF-8 byte instead of raising. Under GPT-2, ß is one token because its two bytes merged during training, 漢 is three, one per byte, and a Gothic letter is four. Newer vocabularies bought some of that back; o200k_base has 漢 whole. The Gothic letter still costs four everywhere. Second, before BPE runs there is a regex that splits the text into chunks, and merges never cross chunk boundaries. GPT-2’s pattern keeps a leading space attached to the word that follows it, so " token" and "token" are different tokens. That’s why the model treats the start of a line differently from the middle of one, and why a stray space in a prompt template changes the output.

The vocabulary is the training corpus

GPT-2’s vocabulary has 50,257 entries. That number isn’t round because it’s a sum: 256 byte tokens, 50,000 learned merges, and one special token, <|endoftext|>, id 50256. Radford et al. (2019) describe the byte-level design those numbers come out of. The two newer OpenAI encodings I checked, cl100k_base and o200k_base, are 100,277 and 200,019.

The merges encode the training corpus. GPT-2 was trained mostly on English web text, so English words are cheap: " tokenizer" is two tokens. The same concept in Spanish, " tokenizador", is three. " desvanecimiento" is six. A German word with an umlaut, " Wörter", is three; one of those three is the two bytes of ö welded to the r after it. The tokenizer has no notion of a character; it has bytes and frequencies.

Numbers are the case that surprises engineers. GPT-2 tokenizes 1234567 as 123, 45, 67, because that’s what the merge ranks happen to produce. cl100k_base changed the regex to cap digit runs at three, which bounds the damage without fixing it: the cap groups left to right, so 1000 becomes 100 and 0. That’s a tokenizer decision with a direct effect on what the model can compute, and it was made in a regex.

Be fair about why. GPT-2’s merges were learned from frequency alone, and in web text 1000 is far more common than 1, 0, 0, 0; a merge table that respects frequency will swallow it whole. Nobody chose to break arithmetic. The cost only became visible once people started asking these models to add.

The same paragraph costs more in Spanish

I took one paragraph of 184 bytes, wrote it in English and in Spanish, and built a JSON payload of roughly the same size, 182 bytes. Then I encoded all three with the three tokenizers.

Spanish costs about 1.7× English under GPT-2 and still 1.2× under the newest encoding. JSON costs about 2.3× under all three: braces, quotes, and keys do not compress. The 26 quotes in that payload never appear as a token on their own; each one is welded to the brace, colon, or space beside it, in ", ":, ",, {". Twenty-six quotes, twenty-six tokens carrying one.

Switch the sample and the encoding. Newer vocabularies mostly close the Spanish gap; the JSON one does not move.

Three things follow, and all three are billing or capacity problems, not modelling problems. A context window quoted in tokens holds less Spanish than English, and much less JSON. A per-token price is a per-language price. And a prompt template that wraps every request in structured metadata is paying the JSON rate on every call.

Nothing in the type system says so

In software, the parser is code and the model is data. You can change the parser, rebuild, and read the same data. Here the parser is coupled to the weights. The embedding matrix has one row per id, and the meaning of row 11,241 is " token" only because that’s what GPT-2’s merges said when the model was trained. Under cl100k_base the same row is Seg. Encode the same text with a different tokenizer and you get ids that index the same matrix and mean something else. Whether that is loud or silent is luck. The same English paragraph under o200k_base reaches id 99,665, which a 50,257-row matrix refuses. Go the other way, GPT-2 ids into a model with a larger vocabulary, and every id lands. Nothing throws. Even the refusal is softer than it looks: production models pad the embedding to a multiple of 64 or 128 for tensor-core alignment, so the matrix is wider than the vocabulary, and on a GPU an out-of-range index surfaces as an asynchronous assert at some unrelated line rather than where it happened.

This is what makes the tokenizer part of the artifact. In GPT-2 that same matrix is also the output head, so a tokenizer swap corrupts the lookup on the way in and the sampling distribution on the way out, from one array of weights. The model file and the tokenizer file are two halves of one thing, and the failure mode when they drift is silent: the model produces fluent text about the wrong input.

What the ML engineer watches

Four things to check before any of this reaches production.

Tokenizer and model versioned together. Store the tokenizer’s hash next to the weights and check it at load time. A mismatch that doesn’t fail on load is the worst kind.

Cost per language and per domain. Measure it on your real traffic before you estimate a bill or promise a context length. The 8k window is 8k English tokens; for the JSON payload above it is closer to 3.5k of the same content.

Special tokens. tiktoken.encode raises by default if the text contains something that looks like <|endoftext|>, precisely because a user could type it. That default is a security decision; check what your serving layer does with it, and where your chat template inserts its own.

Length in tokens, not characters. Every limit the model has is in tokens. Truncation, context budgeting, and rate limits all need the tokenizer in the request path, not a character count.

None of this is modelling work. It’s the part of the model that behaves most like ordinary software, which is exactly why it ships without review: a function that turns strings into integers looks like preprocessing right up until the moment it isn’t.