Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 5 · Modern AI

Predicting the next token

A language model does one job: given the text so far, it gives a probability to every possible next token. Build one from counts, type a few letters to see its guesses, and make it write.

About 10 minutes · Every count, probability and bit is computed in your browser; the formulas are in The maths

Overview

A language model reads text as tokens, as in Tokens: text as numbers, and predicts the next one. It does not pick a single answer: it gives every token in the vocabulary a probability. Writing text is then repeated guessing: pick a token from those probabilities, add it to the text, and ask again.

To keep every number checkable, the tokens here are single characters and the model is the simplest one there is: it counts which character came next in a training text. The text describes the small world invented for this level: eight made-up people, the food each one likes and the place each one lives. Its facts are rules of this page, not claims about anyone. The last lesson, Embeddings: words as points, used the same world.

You'll learn

  • that a language model gives a probability to every possible next token, and writes by sampling from them one at a time
  • how temperature reshapes those probabilities, from always taking the likeliest token to near-random
  • how cross-entropy on held-out text measures a model, and why longer contexts help until the counts run out
  1. The job

    Given the text so far, give a probability to every possible next character.

    The text so far is "mia li". This model looks only at its last 3 characters, "␣li" (␣ marks a space). That piece is called the context, and its length is n. The bars give the probability of each of the 28 symbols coming next: the letters, the space and the full stop.

    In the training text, "␣li" begins likes, lives and limes, so k, v and m get almost all of the probability, and the 28 bars add up to 1. The model never sees "mia": with n = 3 the name is outside the context. From step 3 on, a slider changes n.

    Large language models usually predict subword tokens, from vocabularies of tens of thousands of tokens or more. Here a token is one character, a simplification that keeps every count small enough to show.

    The maths

    Write the text as tokens x1,x2,…x_1, x_2, \dots from the vocabulary V\mathcal{V}, here the 28 symbols, so ∣V∣=28|\mathcal{V}| = 28. A model with context length n gives, at each position t,

    p(xt=v∣xt−n,…,xt−1)  for every v∈V,∑v∈Vp(xt=v∣xt−n,…,xt−1)=1.p(x_t = v \mid x_{t-n}, \dots, x_{t-1}) \ \text{ for every } v \in \mathcal{V}, \qquad \sum_{v \in \mathcal{V}} p(x_t = v \mid x_{t-n}, \dots, x_{t-1}) = 1.

    Looking only at the last n tokens is the Markov assumption behind n-gram models. Here n is the context length; the usual name counts the whole run of n + 1 characters, so this is an (n + 1)-gram model. It is a simplification: the next character can depend on text further back, as the name does here.

  2. Counting

    The probabilities are counts divided by totals, with a little added so nothing is impossible.

    Counting through the training text of 2,000 sentences (42,030 characters), the context "␣li" occurs 1,699 times. It is followed by k 830 times, by v 782 times and by m 87 times. The bars now show those counts.

    So the probability of k is about 830 out of 1,699: 0.488. Each count gets a small extra k = 0.01 before dividing (add-k smoothing), so a character never seen after "␣li" gets 0.0000059, not 0. Here k is the smoothing constant, not the letter k. Without it, a context followed in other text by a character never seen after it would get probability 0 for that character, and the measure in step 5 would be infinite.

    A context that never occurs in the training text has all its counts 0, so the formula gives every character the same probability, 1/28. Counting has nothing to say about it.

    The maths

    With c(context,v)c(\text{context}, v) the number of times v followed the context in the training text, and c(context)=∑vc(context,v)c(\text{context}) = \sum_{v} c(\text{context}, v),

    p(v∣context)=c(context,v)+kc(context)+k ∣V∣,k=0.01.p(v \mid \text{context}) = \frac{c(\text{context}, v) + k}{c(\text{context}) + k\,|\mathcal{V}|}, \qquad k = 0.01.

    For the context "␣li": p(k∣context)=(830+0.01)/(1699+0.01×28)≈0.488p(\mathrm{k} \mid \text{context}) = (830 + 0.01) / (1699 + 0.01 \times 28) \approx 0.488. Adding k to each of the 28 counts adds k ∣V∣k\,|\mathcal{V}| to the total, so the probabilities still add up to 1. With all counts 0 the formula gives k/(k ∣V∣)=1/28k / (k\,|\mathcal{V}|) = 1/28. With k = 0 these are the maximum-likelihood estimates: the plain shares of the counts.

  3. Your context

    Type in the box under the picture. The bars show the model's probabilities for the character after your text.

    The outlined characters are the context the model uses: the last n known characters. The context slider sets n from 1 to 8. Try "zoe likes ", then "zoe li", then a few letters that never appear in the invented world.

    Capitals are turned into lower case, and the box takes up to 80 characters. A character outside the 28 symbols, such as a digit, a comma or an emoji, is shown as a dashed chip with a ? and skipped: the context is made of the known characters before it. If the context never occurred in training, or there are fewer than n known characters, every character gets 1/28. What you type stays in this page; it is not stored or sent anywhere.

    The maths

    The probabilities come from the formula in step 2, with the context taken from your text. With n = 1 the context is a single character, so the model can only learn which letters tend to follow which. Each extra character of context splits the counts into more, smaller groups.

  4. Generate

    To write, sample one character from the probabilities, add it to the text, and repeat. Temperature reshapes the probabilities first.

    Under the picture are 80 characters generated after your text at each of four temperatures, with n from the context slider. Each press of Generate uses a new seed for the random draws, so the samples change. The temperature slider picks which one the bars show: filled bars at that temperature, dashed outlines at τ = 1.

    At τ = 1 the model samples from its own probabilities. A smaller τ sharpens them towards the likeliest character; a larger τ flattens them. At τ = 0 it always takes the likeliest character (greedy). If two characters tie for the top, as h and l do after "he␣" here (199 times each), one of them is picked at random.

    Over many presses (at n = 3, from the starting text), the τ = 0 text keeps to a few words and repeats them. As τ rises the samples vary more, and at τ = 1.5 more of the words are not words of the invented world, because unlikely characters are picked more often.

    The maths

    Temperature τ is applied to the log-probabilities with the softmax from Embeddings: words as points:

    qτ(v)=softmax⁡ ⁣(log⁡p(v)τ)=p(v)1/τ∑u∈Vp(u)1/τ.q_\tau(v) = \operatorname{softmax}\!\left(\frac{\log p(v)}{\tau}\right) = \frac{p(v)^{1/\tau}}{\sum_{u \in \mathcal{V}} p(u)^{1/\tau}}.

    τ = 1 gives back p. As τ→0\tau \to 0, all the probability goes to the largest p(v) (split evenly if several tie): greedy choice. As τ→∞\tau \to \infty, every p(v)1/τ→1p(v)^{1/\tau} \to 1 and q becomes uniform. Each new character is drawn from qτq_\tau by one uniform random number and the cumulative sums of qτq_\tau.

  5. Longer context

    A longer context helps, as long as the training text contains enough of them. The bars measure the model on text it never counted.

    The held-out text is 300 new sentences from the invented world, made with a different seed and never used for counting. For every character in it, the model gives the true next character some probability p, and we take −log⁡2p-\log_2 p: 0 bits for a sure, right guess, more the less probability it gave. The average is the cross-entropy, in bits per character. A uniform guess, 1/28 for everything, costs 4.81 bits.

    With the full training text, the held-out bits fall from 1.99 at n = 1 to 0.47 at n = 3, after which the gains are small: 0.31 at n = 8. No context in the held-out text is unseen. The eight lines under the picture are generated at n = 1 to 8 with τ = 1. Over many presses, from the starting text, at n = 1 most of the complete words are not words of the invented world; from n = 3 on nearly all of them are.

    Now press Small training text: 50 sentences. The held-out bits are lowest at n = 3, 0.59, and rise at every n after that, to 1.32 at n = 8, while the training bits stay low (0.33 at n = 8). The row under the bars gives the share of held-out contexts never seen in training: 15.3% at n = 8.

    The maths

    The cross-entropy over the N positions of the held-out text that have n characters before them is

    H=−1N∑tlog⁡2p(xt∣xt−n,…,xt−1).H = -\frac{1}{N} \sum_{t} \log_2 p(x_t \mid x_{t-n}, \dots, x_{t-1}).

    It is the many-class form of the log-loss from logistic regression, averaged over positions, in base 2 so it reads as bits. The uniform guess gives log⁡228≈4.81\log_2 28 \approx 4.81 bits. An unseen context gets the uniform guess, so it costs exactly log⁡228\log_2 28 bits; with the small text at n = 8, those 15.3% of positions alone add 15.3% × 4.81 = 0.74 bits per character to the average.

  6. Why counting runs out

    Each extra character of context multiplies the number of possible contexts by 28, and counts learn nothing about a context they have not seen.

    There are 28828^8 = 377,801,998,336 possible 8-character contexts. The full training text has 731 of them. That is enough here because the invented world is tiny and regular: 26 different words in three sentence patterns. So counting does well on it, and the solid line is lowest at n = 8.

    The small text shows what happens otherwise. With longer contexts more of the held-out contexts were never counted, each costs the full uniform guess, and the dashed line turns up after n = 3. Real text is far less regular than this world, so with long enough contexts most contexts in new text have never been seen, even in a very large training text.

    Counts treat every context on its own: with n = 7, seeing "mia lik" many times teaches nothing about "zoe lik", although an e comes next in both. Neural language models address this by sharing what they learn across similar contexts, for example through embeddings like those of the last lesson, so a context they never saw can still get sensible probabilities. The ones that write text are trained on the same job, predicting the next token with cross-entropy as the loss, at far larger scale. The next lessons build the parts of one, starting with attention.

    The maths

    With ∣V∣|\mathcal{V}| symbols there are ∣V∣n|\mathcal{V}|^n possible contexts of length n, and each needs its own row of ∣V∣|\mathcal{V}| counts. A training text of M characters fills at most M of those rows, so once ∣V∣n|\mathcal{V}|^n is far larger than M almost all rows are empty.

Quick check

What does temperature 0 do?

Right. All the probability goes to the largest one. That is why the τ = 0 samples keep repeating a few words; a tie, such as h and l after "he␣", is split at random.

That is the other end: as τ grows very large the probabilities flatten towards uniform. τ = 0 sharpens them completely.

That is τ = 1. Temperature 0 keeps only the likeliest token.

Temperature changes only how the next token is picked from the probabilities. The counts, and so the model, stay the same.

Why does held-out cross-entropy matter more than training cross-entropy?

Right. With the small text at n = 8 the training text costs 0.33 bits per character but new text costs 1.32: the long contexts memorise the 50 sentences. The same idea as the test error in earlier levels.

It is usually higher, not lower: at n = 8 with the small text, 1.32 held-out against 0.33 on the training text.

Speed is not the reason. Held-out text is used because it was not counted, so it shows how the model does on new text.

They can be close, as with the full text here, but with the small text and long contexts they are far apart.

Lesson complete

  • A language model gives a probability to every possible next token; it writes by sampling one token, adding it, and repeating.
  • Temperature τ reshapes the probabilities: τ = 1 is the model itself, τ = 0 always takes the likeliest token, larger τ flattens towards uniform.
  • Cross-entropy on held-out text measures a model in bits per token. Longer contexts help until more and more of the contexts in new text were never seen in training; each of those costs the full uniform guess.

Common misconceptions

  • "More context is always better." With the small text, held-out bits rise after n = 3, from 0.59 to 1.32, because more and more contexts were never counted.
  • "The model picks the right next character." It gives probabilities; a sampled text is one random draw from them, and a different seed gives a different text.
  • "Temperature changes what the model learned." It only changes how the next token is drawn from the same probabilities.
Next lesson: Attention →

Further reading

  1. Jurafsky & Martin, Speech and Language Processing (3rd ed. draft), Chapter 3, "N-gram Language Models".

About the data. The world is invented: eight made-up people (ana, ben, eli, mia, noa, sam, tom, zoe), six foods and four places, with a fixed table of who likes which food and lives where. Its facts are rules of this page, not claims about anyone. All the text is made in your browser by the mulberry32 random number generator: the training text is 2,000 sentences from seed 261, the held-out text 300 sentences from seed 262, and the small training text 50 sentences from seed 264. For each sentence, one uniform draw u and then a person chosen uniformly: if u < 0.4 the sentence is "<person> likes <their food>.", if u < 0.8 it is "<person> lives by/in the <their place>.", and otherwise "<person> and <friend> are friends.", with the friend drawn uniformly until it differs from the person. The sentences are joined with single spaces. The counts use add-k smoothing with k = 0.01. Generated text starts at seed 263, and each press of Generate adds 1 to the seed.

If this helped, you can .