Before the transformer reads a sentence, another learned object has already made a decision.

It decides which spans are familiar enough to name.

That object is the tokenizer.

We often describe tokenization as preprocessing: text goes in, token ids come out, and the model begins. As an engineering boundary, that is useful. As a mental boundary, it hides too much.

A tokenizer is a phrasebook learned from a corpus. Its vocabulary says which strings have compact names. Its merge table says which adjacent patterns were common enough to be fused. When a future prompt arrives, the tokenizer does not only parse it. It asks whether the phrasebook already knows how to say it.

same characters
different merge table
different sequence length
different route through the model

That is the sense in which the merge table is a prior. It is not a prior over truth, and it is not a full semantic theory of language. It is a prior over which strings get compact coordinates.

How the Phrasebook Is Written

Byte pair encoding started life as a compression algorithm. Philip Gage’s 1994 algorithm repeatedly replaced a frequent adjacent byte pair with a new symbol, recording a table so the original data could be reconstructed.1 The principle is local:

find a frequent pair
merge it
repeat

For compression, this is a way to trade a dictionary for a shorter message. For language models, the dictionary becomes an embedding table, and the shorter message becomes a shorter sequence of neural computation.

Sennrich, Haddow, and Birch adapted BPE-style subword units to neural machine translation to address the open-vocabulary problem.2 A word-level translation system has an awkward cliff: if a word is outside the vocabulary, the system needs a backoff mechanism. Subwords soften the cliff. Unknown names, compounds, cognates, and morphology can be represented by smaller pieces.

SentencePiece pushed the preprocessing boundary again by training subword models directly from raw sentences instead of assuming a separate pre-tokenized word stream.3 That matters because whitespace, punctuation, and language-specific tokenization rules are not neutral either. If the point is an end-to-end text model, the text boundary itself deserves suspicion.

The success of subword tokenization is real. It made fixed-vocabulary neural models more robust to rare words and open text. But every phrasebook has a shape. A BPE tokenizer can spell any string from small pieces, but it does not spell every string with the same number of pieces.

The choice of phrasebook also matters beyond raw compression. Bostrom and Durrett compared BPE with unigram language-model tokenization for masked language-model pretraining and found that unigram tokenization often aligned better with morphology while avoiding some failures of BPE’s greedy construction.4 That is a useful warning for this post: the lab is about BPE because BPE is inspectable, not because BPE is the only possible subword prior.

The corpus that trained the tokenizer leaves a pronunciation guide for future text.

What a Merge Buys

Suppose a raw-text BPE trainer starts with characters as symbols. At step (k), it chooses the adjacent pair

\[(a^\*, b^\*) = \arg\max_{(a,b)} c_k(a,b),\]

where (c_k(a,b)) is the count of that pair under the current segmentation. Then all non-overlapping appearances of (a^* b^*) are replaced by a new symbol.

Ignoring overlap details, a merge with count (c) saves about (c) tokens on the training corpus. The early merges usually buy obvious fragments: spaces with common letters, suffixes, punctuation patterns, pieces of frequent words. Later merges buy more specialized phrases. The merge budget is finite, so every phrase admitted to the book displaces another possible phrase. Spending a rank on key_cache means not spending it on something else.

This turns tokenization into a quiet selection policy:

the reference corpus chooses which future strings get short names

For dense attention, shortening a sequence from (n) to (m) changes the attention comparison count from roughly (n^2) to (m^2). In autoregressive generation, it also changes how many decoding steps are needed to emit the same characters. During training, it changes where the loss is applied, which embeddings receive updates, and how strings are broken across positions.

None of this means a tokenizer determines what a model can understand. Neural networks are annoyingly good at learning around bad interfaces. But an interface can still make one part of the world direct and another part circuitous.

Numbers, code identifiers, logs, names, romanized words, low-resource languages, and domain-specific notation are exactly where this matters. They are often compositional to a human and fragmented to a tokenizer trained elsewhere.

Lab: Two Phrasebooks, One Probe

The lab below trains two tiny BPE tokenizers in the browser. Both use the same merge budget and the same greedy algorithm. The only difference is the reference corpus.

The general tokenizer is trained on a mostly prose corpus. The domain tokenizer is trained on a mixture you control: more code-like rows, more number/log rows, or a balanced blend. Then both tokenizers encode the same probe string.

This is not a production tokenizer. It is deliberately smaller and more inspectable: raw text, deterministic training rows, character initialization, greedy pair merges. The point is to make one mechanism visible. Change the reference distribution, and the same future text receives a different spelling. The lab’s exported audit checks that merge encoding preserves the original string, merge budgets are respected, larger merge budgets do not increase the weighted corpus token count, and domain-heavy corpora shorten matching probes.

General tokenizer Domain tokenizer Merge count curve

General tokenizer pieces

Domain tokenizer pieces

First domain merges

    Deterministic toy experiment. The dense attention proxy is the selected probe's domain-token count squared divided by the general-token count squared. Lower than 1 means the domain tokenizer made this exact string cheaper under that crude compute proxy.

    At the default setting, the code probe goes from 42 pieces under the general tokenizer to 33 under the domain tokenizer. The crude dense-attention proxy drops to about 0.62x. The market log only moves from 50 to 48 pieces, and the rare string stays at 41 either way. The merge table helped where the reference corpus had learned relevant fragments. It did not become a universal phrasebook.

    Try three inspections.

    Raise Code share and keep the code probe selected. The general tokenizer can spell attention_scores and key_cache, but it spends many smaller pieces. The domain tokenizer starts naming underscores, suffixes, and identifier fragments. The same string needs fewer syllables.

    Now switch to the market-log probe and raise Number/log share. The advantage moves. Date fragments, decimal patterns, and field-name syntax become worth merging. Push the corpus toward code again and the market string may fragment more than it did under the general tokenizer. A fixed phrasebook cannot make every dialect compact.

    Finally, use the rare-string probe. It does not become magically efficient just because the tokenizer has seen “technical” text. A BPE table generalizes through shared adjacent fragments, not through understanding. If the important pattern is not frequent enough to buy merges, it falls back to smaller units.

    Token Count Is Only the Obvious Symptom

    Token count is the easiest thing to measure, so it is the thing the lab shows. But the representational effect is broader.

    An embedding table gives each token id a learned vector. A frequent token has a single coordinate that can accumulate evidence across many contexts. A rare string split into five pieces has to be reconstructed from smaller coordinates and positions. That can be good if the pieces are meaningful. It can be bad if the split cuts across the structure that mattered.

    Consider identifiers. retry_count is not just the characters r e t r y _ c. In code, the underscore is a boundary, retry is a concept, and count is a role. A tokenizer trained on prose may see the underscore as punctuation noise. A tokenizer trained on code may spend merges that preserve the idiom. Neither choice is universally correct. The question is whether the phrasebook matches the task distribution.

    Numbers are worse in a different way. A decimal string is simultaneously text, syntax, and magnitude. Splitting it into arbitrary chunks can make arithmetic look like spelling. Some models can learn the patterns anyway, but the tokenizer has changed the path. It decides which chunks receive atomic embeddings, where positional boundaries fall, and how many recurrent decoding decisions are needed to emit or inspect the number. Singh and Strouse’s arithmetic study makes this concrete: changing number tokenization direction changed frontier-model arithmetic behavior, and some error patterns followed the tokenization rather than the mathematical expression alone.5

    This is why tokenization audits should be empirical. For a workload you care about, measure:

    1. how many tokens representative examples use;
    2. where meaningful boundaries are cut;
    3. which rare-but-important strings fragment badly;
    4. how much the answer changes under another tokenizer or byte-level baseline.

    The important comparison is not “does the tokenizer work on average?” It is “which parts of my distribution are being forced to spell themselves out?”

    Refusing the Phrasebook

    One response is to remove the learned vocabulary boundary altogether.

    CANINE operates directly on character sequences and uses downsampling to make the finer-grained input efficient enough for a deep encoder.6 ByT5 asks a similar question at the byte level: can a mostly standard Transformer process raw UTF-8 bytes and avoid the preprocessing artifact? Its authors emphasize the tradeoff directly: byte or character sequences are longer, but byte-level models can be competitive and more robust on tasks sensitive to noise, spelling, and pronunciation.7

    More recent tokenizer-free or tokenizer-light proposals keep circling the same pressure point. T-FREE, for example, represents words through sparse activation patterns over character triplets, aiming to avoid a reference-corpus tokenizer while reducing the embedding and output-head burden.8

    The moral is not that subword tokenizers are obsolete. They remain a very good engineering compromise. A byte-level model may pay with longer sequences; a subword model may pay with reference-corpus assumptions. The right question is not whether there is a free representation. There is not.

    The right question is which assumptions you want to make visible.

    Provenance Questions

    It is useful to treat a tokenizer the way we treat a training set, a benchmark, or a labeling policy: as a dataset artifact with assumptions.

    Ask where it came from. Ask what it makes cheap. Ask what it fragments. Ask whether the target users write in the same style, language, notation, and domain as the reference corpus. Ask whether your evaluation reports characters, words, or tokens when comparing systems.

    This matters even when model weights are frozen. Retrieval chunks are usually budgeted in tokens. Prompt templates are budgeted in tokens. Tool traces, source code, logs, citations, tables, and long-context documents are all routed through a tokenizer before the model sees them. A ten percent token-count difference is not an aesthetic detail when the context window, attention kernel, cache, and billing system are all token-indexed.

    The most dangerous phrase is “just preprocessing.”

    Preprocessing is where the world is discretized. Discretization is a model.

    The merge table remembers a corpus. Then every future string negotiates with that memory, one piece at a time.

    Primary Sources

    1. Philip Gage, “A New Algorithm for Data Compression,” The C Users Journal, February 1994. HTML mirror, Semantic Scholar. ↩

    2. Rico Sennrich, Barry Haddow, and Alexandra Birch, “Neural Machine Translation of Rare Words with Subword Units,” ACL 2016. ACL Anthology, arXiv. ↩

    3. Taku Kudo and John Richardson, “SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing,” EMNLP 2018 System Demonstrations. ACL Anthology, arXiv. ↩

    4. Kaj Bostrom and Greg Durrett, “Byte Pair Encoding is Suboptimal for Language Model Pretraining,” Findings of EMNLP 2020. ACL Anthology, arXiv. ↩

    5. Aaditya K. Singh and DJ Strouse, “Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs,” arXiv 2024. arXiv. ↩

    6. Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting, “Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation,” TACL 2022. ACL Anthology, arXiv. ↩

    7. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel, “ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models,” TACL 2022. ACL Anthology, arXiv. ↩

    8. Bjorn Deiseroth, Manuel Brack, Patrick Schramowski, Kristian Kersting, and Samuel Weinbach, “T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings,” EMNLP 2024. ACL Anthology, arXiv. ↩