ML Explorer · Level 5 · Modern AI
Tokens: text as numbers
A model reads text as a list of numbers. Watch a tokeniser build its vocabulary by merging the commonest pair of pieces, again and again, and type a sentence of your own to see it split.
Overview
Every model so far took numbers in. Text is not numbers, so before a language model sees a sentence, the sentence is cut into pieces called tokens, and each token is replaced by its number in a fixed list. This lesson builds such a list from a small text, live.
The text describes a small world invented for this level: eight made-up people, the food each one likes and the place each one lives. Its facts are rules of this page, not claims about anyone. The next lessons use the same world.
You'll learn
- that a model sees text as a sequence of token ids from a fixed vocabulary
- how byte-pair encoding builds that vocabulary by merging the most frequent adjacent pair, one merge at a time
- the trade-off: a bigger vocabulary gives shorter sequences
-
Text is characters
The simplest way to turn text into numbers: one token per character, and each character gets an id.
The training text is 400 sentences from the invented world, such as "eli likes plums." and "ana and noa are friends.", joined with spaces. Its facts are rules of this page. Cut into characters, it is 8,541 tokens long.
Under the picture are the first two sentences, one chip per character, with each character's id underneath; ␣ marks a space. The bars count how often each id occurs in the whole text. The space is the most common. The letters c, j, q and x never occur, but they are still on the list, so text that uses them can be written in ids too.
The maths
After tokenising, a text is a sequence of ids , here with = 8,541. Each is a whole number from 0 to 27. A model never sees the letters, only this sequence.
-
A fixed list
The list of every token the model can use is its vocabulary, written . Here it holds 28 symbols: space, full stop and the 26 letters, with ids 0 to 27.
An id is a label, not an amount: id 13 (l) is not "more" than id 2 (a), and the order of the list means nothing. So a model does not use the id as a number directly. It turns id into a one-hot vector : a list of 28 numbers, all 0 except a 1 in entry , counting entries from 0 like the ids. A vector is an ordered list of numbers, as in data as numbers.
Move the character slider to pick a character of the first sentence and see its one-hot vector. Every character gets a different vector, and no two are more alike than any other two.
The maths
With vocabulary size ,
For a matrix with one row per token, is row of , so a one-hot vector is the same as looking up a row of a table. The next lesson learns such a table.
-
Merge the commonest pair
Byte-pair encoding grows the vocabulary one token at a time: find the pair of neighbouring tokens that occurs most often, and make it a new token.
Play steps through the lesson. Merge once does a single merge; Train keeps merging, one every 0.4 seconds, up to 60 merges (with reduced motion it goes straight to the end); Reset goes back to characters. Working out the merges from a text is called training the tokeniser. It is counting, and the same text always gives the same merges.
Each merge counts every pair of neighbouring tokens inside a word (a pair never spans a space or a full stop), takes the most frequent pair, replaces every occurrence of it in the text by one new token, and adds that token to the list with the next free id, 28, 29 and so on. A tie goes to the pair that occurs first in the text.
The first merge is l + i, which occurs 389 times (in likes, lives, limes and eli); the second is e + s, 386 times. Merges build on earlier merges: the word friends becomes a single token at merge 19. Watch the chips under the picture fuse as the merges arrive.
The maths
Write for the number of times token is followed by token inside a word, counted left to right without overlaps. Each merge picks
with ties broken by first occurrence, then replaces every one of those occurrences by the single token . Each replacement turns two tokens into one, so the text gets shorter by exactly the number of replacements:
where is the token count after merges. Overlaps matter only when : in "aaa" the pair "aa" fits twice but can be replaced only once, so it is counted once, scanning left to right, which is how the replacement is done. The new token from merge (counting from 1) gets id .
-
The trade-off
Every merge adds one token to the vocabulary and makes the text shorter. The line shows both, for every merge from 0 to 60.
After 60 merges the vocabulary has 88 tokens and the text is 3,853 tokens long, down from 8,541: 2.22 characters per token. Move the merges slider to read any point.
The savings shrink: the first merge saves 389 tokens and the 60th saves 17, and here no merge saves more than the one before it. This text allows 63 merges in all; after that every word is one token, the text is 3,811 tokens, and no pair occurs twice. At 60 merges only figs is still in pieces.
Why not merge all the way, or further? A model keeps a row of numbers for every token in its vocabulary, so a bigger vocabulary means more to learn, and rare tokens are seen too seldom to learn well. A smaller one means longer sequences, so more steps for every text. Tokenisers pick a size in between.
The maths
The compression ratio is
It starts at 1, with one token per character. After a merge, no pair is more common than the pair just merged: every occurrence of a new pair such as contains its own occurrence of , so it occurs at most times, and the counts of other pairs can only stay the same or fall. That is why each merge saves no more than the one before.
-
Your text
Type a sentence in the box under the picture. It is split with the merges so far, and each token is shown with its id.
Capitals are turned into lower case, and the box takes up to 80 characters. A character outside the 28 symbols, such as a digit, a comma or an emoji, has no id: it is shown as a dashed chip with a ?. What you type stays in this page; it is not stored or sent anywhere.
The merges slider sets how many merges are used. With all 60, the starting sentence shows what happens to words the tokeniser never saw: like becomes lik + e, because the text only ever has likes, and kiwis is spelt out letter by letter. Nothing is lost: any word made of the 28 symbols can be written with the pieces the vocabulary has.
The maths
A new text is tokenised by starting from its characters and applying the learned merges in the order they were learned, each one to every non-overlapping occurrence, left to right. Joining the tokens back together gives the text exactly.
-
Real tokenisers
Many tokenisers used with large language models learn merges the same way, at far larger scale: from far more text, with vocabularies of tens of thousands of tokens or more.
Many start from bytes instead of characters. Stored text is a sequence of bytes, each a number from 0 to 255; in the common UTF-8 encoding every character takes 1 to 4 bytes. The 256 byte values are the starting vocabulary, so any text, in any script, can be written in tokens, and nothing is ever unknown. The chips now show your text as bytes: each of our 28 symbols is one byte, and a character such as é or an emoji takes several.
Sennrich, Haddow & Birch adapted byte-pair encoding, first a method for compressing data, to split rare words into subword units for machine translation. Once the tokens are fixed, the model works only with their ids. The next lesson gives each token a learned vector instead of a one-hot one.
The maths
With bytes as the base, after merges, and every text can be encoded, because every text is a sequence of bytes.
Quick check
Lesson complete
- A model sees text as a sequence of token ids from a fixed vocabulary ; an id becomes a one-hot vector .
- Byte-pair encoding starts from single symbols and repeatedly merges the most frequent neighbouring pair; each merge shortens the text by the number of replacements and adds one token.
- A bigger vocabulary gives shorter sequences: here 88 tokens give 2.22 characters per token. Pieces of words let a tokeniser write words it never saw.
Common misconceptions
- "A token is a word." Tokens are whatever pieces the merges built: after 60 merges, figs is still several tokens, and like is lik + e.
- "Ids that are close are similar." An id is only a place in the list. Every pair of one-hot vectors is equally far apart.
- "The tokeniser is trained together with the model." Byte-pair merges are found by counting, before the model is trained, and then stay fixed.
Further reading
- Jurafsky & Martin, Speech and Language Processing (3rd ed. draft), Chapter 2, "Words and Tokens".
- Sennrich, Haddow & Birch, "Neural Machine Translation of Rare Words with Subword Units", arXiv 1508.07909 (2015; ACL 2016), which adapted byte-pair encoding to split rare words into subword units.
About the data. The world is invented: eight made-up people (ana, ben, eli, mia, noa, sam, tom, zoe), six foods and four places, with a fixed table of who likes which food and lives where. Its facts are rules of this page, not claims about anyone. The training text is 400 sentences made in your browser by the mulberry32 random number generator, seed 241. For each sentence, one uniform draw u and then a person chosen uniformly: if u < 0.4 the sentence is "<person> likes <their food>.", if u < 0.8 it is "<person> lives by/in the <their place>.", and otherwise "<person> and <friend> are friends.", with the friend drawn uniformly until it differs from the person. The sentences are joined with single spaces. The merges are found from this text by the rule in step 3, pairs inside words only, ties to the pair that occurs first.
If this helped, you can .