Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 5 · Modern AI

Tokens: text as numbers

A model reads text as a list of numbers. Watch a tokeniser build its vocabulary by merging the commonest pair of pieces, again and again, and type a sentence of your own to see it split.

About 9 minutes · Every count, merge and id is computed in your browser; the formulas are in The maths

Overview

Every model so far took numbers in. Text is not numbers, so before a language model sees a sentence, the sentence is cut into pieces called tokens, and each token is replaced by its number in a fixed list. This lesson builds such a list from a small text, live.

The text describes a small world invented for this level: eight made-up people, the food each one likes and the place each one lives. Its facts are rules of this page, not claims about anyone. The next lessons use the same world.

You'll learn

  • that a model sees text as a sequence of token ids from a fixed vocabulary
  • how byte-pair encoding builds that vocabulary by merging the most frequent adjacent pair, one merge at a time
  • the trade-off: a bigger vocabulary gives shorter sequences
  1. Text is characters

    The simplest way to turn text into numbers: one token per character, and each character gets an id.

    The training text is 400 sentences from the invented world, such as "eli likes plums." and "ana and noa are friends.", joined with spaces. Its facts are rules of this page. Cut into characters, it is 8,541 tokens long.

    Under the picture are the first two sentences, one chip per character, with each character's id underneath; ␣ marks a space. The bars count how often each id occurs in the whole text. The space is the most common. The letters c, j, q and x never occur, but they are still on the list, so text that uses them can be written in ids too.

    The maths

    After tokenising, a text is a sequence of ids x1,x2,…,xNx_1, x_2, \dots, x_N, here with NN = 8,541. Each xtx_t is a whole number from 0 to 27. A model never sees the letters, only this sequence.

  2. A fixed list

    The list of every token the model can use is its vocabulary, written V\mathcal{V}. Here it holds 28 symbols: space, full stop and the 26 letters, with ids 0 to 27.

    An id is a label, not an amount: id 13 (l) is not "more" than id 2 (a), and the order of the list means nothing. So a model does not use the id as a number directly. It turns id ii into a one-hot vector ei\mathbf{e}_i: a list of 28 numbers, all 0 except a 1 in entry ii, counting entries from 0 like the ids. A vector is an ordered list of numbers, as in data as numbers.

    Move the character slider to pick a character of the first sentence and see its one-hot vector. Every character gets a different vector, and no two are more alike than any other two.

    The maths

    With vocabulary size ∣V∣=28|\mathcal{V}| = 28,

    ei=(0,…,0,1entry i,0,…,0)∈R∣V∣,i=0,1,…,∣V∣−1\mathbf{e}_i = (0, \dots, 0, \underset{\text{entry } i}{1}, 0, \dots, 0) \in \mathbb{R}^{|\mathcal{V}|}, \qquad i = 0, 1, \dots, |\mathcal{V}| - 1

    For a matrix W\mathbf{W} with one row per token, eiTW\mathbf{e}_i^{\mathsf{T}}\mathbf{W} is row ii of W\mathbf{W}, so a one-hot vector is the same as looking up a row of a table. The next lesson learns such a table.

  3. Merge the commonest pair

    Byte-pair encoding grows the vocabulary one token at a time: find the pair of neighbouring tokens that occurs most often, and make it a new token.

    Play steps through the lesson. Merge once does a single merge; Train keeps merging, one every 0.4 seconds, up to 60 merges (with reduced motion it goes straight to the end); Reset goes back to characters. Working out the merges from a text is called training the tokeniser. It is counting, and the same text always gives the same merges.

    Each merge counts every pair of neighbouring tokens inside a word (a pair never spans a space or a full stop), takes the most frequent pair, replaces every occurrence of it in the text by one new token, and adds that token to the list with the next free id, 28, 29 and so on. A tie goes to the pair that occurs first in the text.

    The first merge is l + i, which occurs 389 times (in likes, lives, limes and eli); the second is e + s, 386 times. Merges build on earlier merges: the word friends becomes a single token at merge 19. Watch the chips under the picture fuse as the merges arrive.

    The maths

    Write c(a,b)c(a, b) for the number of times token aa is followed by token bb inside a word, counted left to right without overlaps. Each merge picks

    (a∗,b∗)=arg max⁡(a, b)  c(a,b)(a^*, b^*) = \operatorname*{arg\,max}_{(a,\, b)}\; c(a, b)

    with ties broken by first occurrence, then replaces every one of those c(a∗,b∗)c(a^*, b^*) occurrences by the single token a∗b∗a^*b^*. Each replacement turns two tokens into one, so the text gets shorter by exactly the number of replacements:

    Nr+1=Nr−c(a∗,b∗),∣Vr+1∣=∣Vr∣+1N_{r+1} = N_r - c(a^*, b^*), \qquad |\mathcal{V}_{r+1}| = |\mathcal{V}_r| + 1

    where NrN_r is the token count after rr merges. Overlaps matter only when a=ba = b: in "aaa" the pair "aa" fits twice but can be replaced only once, so it is counted once, scanning left to right, which is how the replacement is done. The new token from merge rr (counting from 1) gets id 27+r27 + r.

  4. The trade-off

    Every merge adds one token to the vocabulary and makes the text shorter. The line shows both, for every merge from 0 to 60.

    After 60 merges the vocabulary has 88 tokens and the text is 3,853 tokens long, down from 8,541: 2.22 characters per token. Move the merges slider to read any point.

    The savings shrink: the first merge saves 389 tokens and the 60th saves 17, and here no merge saves more than the one before it. This text allows 63 merges in all; after that every word is one token, the text is 3,811 tokens, and no pair occurs twice. At 60 merges only figs is still in pieces.

    Why not merge all the way, or further? A model keeps a row of numbers for every token in its vocabulary, so a bigger vocabulary means more to learn, and rare tokens are seen too seldom to learn well. A smaller one means longer sequences, so more steps for every text. Tokenisers pick a size in between.

    The maths

    The compression ratio is

    characters per token=number of charactersNr\text{characters per token} = \frac{\text{number of characters}}{N_r}

    It starts at 1, with one token per character. After a merge, no pair is more common than the pair just merged: every occurrence of a new pair such as (a∗b∗,c)(a^*b^*, c) contains its own occurrence of (a∗,b∗)(a^*, b^*), so it occurs at most c(a∗,b∗)c(a^*, b^*) times, and the counts of other pairs can only stay the same or fall. That is why each merge saves no more than the one before.

  5. Your text

    Type a sentence in the box under the picture. It is split with the merges so far, and each token is shown with its id.

    Capitals are turned into lower case, and the box takes up to 80 characters. A character outside the 28 symbols, such as a digit, a comma or an emoji, has no id: it is shown as a dashed chip with a ?. What you type stays in this page; it is not stored or sent anywhere.

    The merges slider sets how many merges are used. With all 60, the starting sentence shows what happens to words the tokeniser never saw: like becomes lik + e, because the text only ever has likes, and kiwis is spelt out letter by letter. Nothing is lost: any word made of the 28 symbols can be written with the pieces the vocabulary has.

    The maths

    A new text is tokenised by starting from its characters and applying the learned merges in the order they were learned, each one to every non-overlapping occurrence, left to right. Joining the tokens back together gives the text exactly.

  6. Real tokenisers

    Many tokenisers used with large language models learn merges the same way, at far larger scale: from far more text, with vocabularies of tens of thousands of tokens or more.

    Many start from bytes instead of characters. Stored text is a sequence of bytes, each a number from 0 to 255; in the common UTF-8 encoding every character takes 1 to 4 bytes. The 256 byte values are the starting vocabulary, so any text, in any script, can be written in tokens, and nothing is ever unknown. The chips now show your text as bytes: each of our 28 symbols is one byte, and a character such as é or an emoji takes several.

    Sennrich, Haddow & Birch adapted byte-pair encoding, first a method for compressing data, to split rare words into subword units for machine translation. Once the tokens are fixed, the model works only with their ids. The next lesson gives each token a learned vector instead of a one-hot one.

    The maths

    With bytes as the base, ∣V∣=256+r|\mathcal{V}| = 256 + r after rr merges, and every text can be encoded, because every text is a sequence of bytes.

Quick check

What does each BPE merge do?

Right. The first merge here was l + i, 389 times, so the text got 389 tokens shorter and the vocabulary one token bigger.

Nothing is removed: the 28 symbols stay, so any text made of them can still be written. Each merge only adds.

Here pairs are counted only inside words, so a merge never spans a space. It joins pieces of words, such as l and i, and whole words appear only when their pieces have been merged.

The other way round: the vocabulary grows by one and the text gets shorter.

Why not give every word its own token?

Right. With whole words, kiwis would have no token at all. With pieces, it is spelt out with tokens the vocabulary already has.

Whole words give shorter sequences: 3,811 tokens here, against 8,541 characters. The problems are the words not on the list and the size of the list.

88 is just where this lesson stops. Real vocabularies often hold tens of thousands of tokens or more; the cost of a bigger one is more to learn, not a hard limit.

The tokens stay in order either way. The trouble is words that are not on the list.

Lesson complete

  • A model sees text as a sequence of token ids from a fixed vocabulary V\mathcal{V}; an id becomes a one-hot vector ei\mathbf{e}_i.
  • Byte-pair encoding starts from single symbols and repeatedly merges the most frequent neighbouring pair; each merge shortens the text by the number of replacements and adds one token.
  • A bigger vocabulary gives shorter sequences: here 88 tokens give 2.22 characters per token. Pieces of words let a tokeniser write words it never saw.

Common misconceptions

  • "A token is a word." Tokens are whatever pieces the merges built: after 60 merges, figs is still several tokens, and like is lik + e.
  • "Ids that are close are similar." An id is only a place in the list. Every pair of one-hot vectors is equally far apart.
  • "The tokeniser is trained together with the model." Byte-pair merges are found by counting, before the model is trained, and then stay fixed.
Next lesson: Embeddings: words as points →

Further reading

  1. Jurafsky & Martin, Speech and Language Processing (3rd ed. draft), Chapter 2, "Words and Tokens".
  2. Sennrich, Haddow & Birch, "Neural Machine Translation of Rare Words with Subword Units", arXiv 1508.07909 (2015; ACL 2016), which adapted byte-pair encoding to split rare words into subword units.

About the data. The world is invented: eight made-up people (ana, ben, eli, mia, noa, sam, tom, zoe), six foods and four places, with a fixed table of who likes which food and lives where. Its facts are rules of this page, not claims about anyone. The training text is 400 sentences made in your browser by the mulberry32 random number generator, seed 241. For each sentence, one uniform draw u and then a person chosen uniformly: if u < 0.4 the sentence is "<person> likes <their food>.", if u < 0.8 it is "<person> lives by/in the <their place>.", and otherwise "<person> and <friend> are friends.", with the friend drawn uniformly until it differs from the person. The sentences are joined with single spaces. The merges are found from this text by the rule in step 3, pairs inside words only, ties to the pair that occurs first.

If this helped, you can .