ML Explorer · Level 5 · Modern AI
Predicting the next token
A language model does one job: given the text so far, it gives a probability to every possible next token. Build one from counts, type a few letters to see its guesses, and make it write.
Overview
A language model reads text as tokens, as in Tokens: text as numbers, and predicts the next one. It does not pick a single answer: it gives every token in the vocabulary a probability. Writing text is then repeated guessing: pick a token from those probabilities, add it to the text, and ask again.
To keep every number checkable, the tokens here are single characters and the model is the simplest one there is: it counts which character came next in a training text. The text describes the small world invented for this level: eight made-up people, the food each one likes and the place each one lives. Its facts are rules of this page, not claims about anyone. The last lesson, Embeddings: words as points, used the same world.
You'll learn
- that a language model gives a probability to every possible next token, and writes by sampling from them one at a time
- how temperature reshapes those probabilities, from always taking the likeliest token to near-random
- how cross-entropy on held-out text measures a model, and why longer contexts help until the counts run out
-
The job
Given the text so far, give a probability to every possible next character.
The text so far is "mia li". This model looks only at its last 3 characters, "␣li" (␣ marks a space). That piece is called the context, and its length is n. The bars give the probability of each of the 28 symbols coming next: the letters, the space and the full stop.
In the training text, "␣li" begins likes, lives and limes, so k, v and m get almost all of the probability, and the 28 bars add up to 1. The model never sees "mia": with n = 3 the name is outside the context. From step 3 on, a slider changes n.
Large language models usually predict subword tokens, from vocabularies of tens of thousands of tokens or more. Here a token is one character, a simplification that keeps every count small enough to show.
The maths
Write the text as tokens from the vocabulary , here the 28 symbols, so . A model with context length n gives, at each position t,
Looking only at the last n tokens is the Markov assumption behind n-gram models. Here n is the context length; the usual name counts the whole run of n + 1 characters, so this is an (n + 1)-gram model. It is a simplification: the next character can depend on text further back, as the name does here.
-
Counting
The probabilities are counts divided by totals, with a little added so nothing is impossible.
Counting through the training text of 2,000 sentences (42,030 characters), the context "␣li" occurs 1,699 times. It is followed by k 830 times, by v 782 times and by m 87 times. The bars now show those counts.
So the probability of k is about 830 out of 1,699: 0.488. Each count gets a small extra k = 0.01 before dividing (add-k smoothing), so a character never seen after "␣li" gets 0.0000059, not 0. Here k is the smoothing constant, not the letter k. Without it, a context followed in other text by a character never seen after it would get probability 0 for that character, and the measure in step 5 would be infinite.
A context that never occurs in the training text has all its counts 0, so the formula gives every character the same probability, 1/28. Counting has nothing to say about it.
The maths
With the number of times v followed the context in the training text, and ,
For the context "␣li": . Adding k to each of the 28 counts adds to the total, so the probabilities still add up to 1. With all counts 0 the formula gives . With k = 0 these are the maximum-likelihood estimates: the plain shares of the counts.
-
Your context
Type in the box under the picture. The bars show the model's probabilities for the character after your text.
The outlined characters are the context the model uses: the last n known characters. The context slider sets n from 1 to 8. Try "zoe likes ", then "zoe li", then a few letters that never appear in the invented world.
Capitals are turned into lower case, and the box takes up to 80 characters. A character outside the 28 symbols, such as a digit, a comma or an emoji, is shown as a dashed chip with a ? and skipped: the context is made of the known characters before it. If the context never occurred in training, or there are fewer than n known characters, every character gets 1/28. What you type stays in this page; it is not stored or sent anywhere.
The maths
The probabilities come from the formula in step 2, with the context taken from your text. With n = 1 the context is a single character, so the model can only learn which letters tend to follow which. Each extra character of context splits the counts into more, smaller groups.
-
Generate
To write, sample one character from the probabilities, add it to the text, and repeat. Temperature reshapes the probabilities first.
Under the picture are 80 characters generated after your text at each of four temperatures, with n from the context slider. Each press of Generate uses a new seed for the random draws, so the samples change. The temperature slider picks which one the bars show: filled bars at that temperature, dashed outlines at τ = 1.
At τ = 1 the model samples from its own probabilities. A smaller τ sharpens them towards the likeliest character; a larger τ flattens them. At τ = 0 it always takes the likeliest character (greedy). If two characters tie for the top, as h and l do after "he␣" here (199 times each), one of them is picked at random.
Over many presses (at n = 3, from the starting text), the τ = 0 text keeps to a few words and repeats them. As τ rises the samples vary more, and at τ = 1.5 more of the words are not words of the invented world, because unlikely characters are picked more often.
The maths
Temperature τ is applied to the log-probabilities with the softmax from Embeddings: words as points:
τ = 1 gives back p. As , all the probability goes to the largest p(v) (split evenly if several tie): greedy choice. As , every and q becomes uniform. Each new character is drawn from by one uniform random number and the cumulative sums of .
-
Longer context
A longer context helps, as long as the training text contains enough of them. The bars measure the model on text it never counted.
The held-out text is 300 new sentences from the invented world, made with a different seed and never used for counting. For every character in it, the model gives the true next character some probability p, and we take : 0 bits for a sure, right guess, more the less probability it gave. The average is the cross-entropy, in bits per character. A uniform guess, 1/28 for everything, costs 4.81 bits.
With the full training text, the held-out bits fall from 1.99 at n = 1 to 0.47 at n = 3, after which the gains are small: 0.31 at n = 8. No context in the held-out text is unseen. The eight lines under the picture are generated at n = 1 to 8 with τ = 1. Over many presses, from the starting text, at n = 1 most of the complete words are not words of the invented world; from n = 3 on nearly all of them are.
Now press Small training text: 50 sentences. The held-out bits are lowest at n = 3, 0.59, and rise at every n after that, to 1.32 at n = 8, while the training bits stay low (0.33 at n = 8). The row under the bars gives the share of held-out contexts never seen in training: 15.3% at n = 8.
The maths
The cross-entropy over the N positions of the held-out text that have n characters before them is
It is the many-class form of the log-loss from logistic regression, averaged over positions, in base 2 so it reads as bits. The uniform guess gives bits. An unseen context gets the uniform guess, so it costs exactly bits; with the small text at n = 8, those 15.3% of positions alone add 15.3% × 4.81 = 0.74 bits per character to the average.
-
Why counting runs out
Each extra character of context multiplies the number of possible contexts by 28, and counts learn nothing about a context they have not seen.
There are = 377,801,998,336 possible 8-character contexts. The full training text has 731 of them. That is enough here because the invented world is tiny and regular: 26 different words in three sentence patterns. So counting does well on it, and the solid line is lowest at n = 8.
The small text shows what happens otherwise. With longer contexts more of the held-out contexts were never counted, each costs the full uniform guess, and the dashed line turns up after n = 3. Real text is far less regular than this world, so with long enough contexts most contexts in new text have never been seen, even in a very large training text.
Counts treat every context on its own: with n = 7, seeing "mia lik" many times teaches nothing about "zoe lik", although an e comes next in both. Neural language models address this by sharing what they learn across similar contexts, for example through embeddings like those of the last lesson, so a context they never saw can still get sensible probabilities. The ones that write text are trained on the same job, predicting the next token with cross-entropy as the loss, at far larger scale. The next lessons build the parts of one, starting with attention.
The maths
With symbols there are possible contexts of length n, and each needs its own row of counts. A training text of M characters fills at most M of those rows, so once is far larger than M almost all rows are empty.
Quick check
Lesson complete
- A language model gives a probability to every possible next token; it writes by sampling one token, adding it, and repeating.
- Temperature τ reshapes the probabilities: τ = 1 is the model itself, τ = 0 always takes the likeliest token, larger τ flattens towards uniform.
- Cross-entropy on held-out text measures a model in bits per token. Longer contexts help until more and more of the contexts in new text were never seen in training; each of those costs the full uniform guess.
Common misconceptions
- "More context is always better." With the small text, held-out bits rise after n = 3, from 0.59 to 1.32, because more and more contexts were never counted.
- "The model picks the right next character." It gives probabilities; a sampled text is one random draw from them, and a different seed gives a different text.
- "Temperature changes what the model learned." It only changes how the next token is drawn from the same probabilities.
Further reading
- Jurafsky & Martin, Speech and Language Processing (3rd ed. draft), Chapter 3, "N-gram Language Models".
About the data. The world is invented: eight made-up people (ana, ben, eli, mia, noa, sam, tom, zoe), six foods and four places, with a fixed table of who likes which food and lives where. Its facts are rules of this page, not claims about anyone. All the text is made in your browser by the mulberry32 random number generator: the training text is 2,000 sentences from seed 261, the held-out text 300 sentences from seed 262, and the small training text 50 sentences from seed 264. For each sentence, one uniform draw u and then a person chosen uniformly: if u < 0.4 the sentence is "<person> likes <their food>.", if u < 0.8 it is "<person> lives by/in the <their place>.", and otherwise "<person> and <friend> are friends.", with the friend drawn uniformly until it differs from the person. The sentences are joined with single spaces. The counts use add-k smoothing with k = 0.01. Generated text starts at seed 263, and each press of Generate adds 1 to the seed.
If this helped, you can .