Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 5 · Modern AI

Attention

Attention lets every position in a sequence look straight back at any earlier one. Drag a query and watch the weights shift, then train a small attention model live on the task a recurrent network found hard.

About 11 minutes · Every weight, run and accuracy is computed in your browser; the formulas are in The maths

Overview

In predicting the next token, a model looked only at the last few characters. In recurrent networks, a network read a sequence one item at a time and had to squeeze everything it might need later into one hidden state. Attention takes another route: at each position, the model compares that position with every earlier one and takes a weighted mix of what it finds there.

The first half of the lesson shows the mechanism with four hand-placed vectors you can move. The second half trains a small attention model on the echo task from recurrent networks: say the bit you saw k steps ago.

You'll learn

  • how attention scores a query against every key with a scaled dot product, and turns the scores into weights with softmax
  • that the result is a weighted average of the values: a soft lookup
  • why a position can reach any earlier position in one step, so distance stops mattering
  1. The problem from Level 4

    The task: a stream of 12 random bits, and at every step, say the bit from k steps ago. Here k = 8, so the target at step 9 is the bit at position 1.

    In recurrent networks, a network read the stream one bit at a time. To answer at step 9 it had to carry bit 1 in its hidden state through 8 updates, alongside everything else it was carrying. Within its budget of 10 epochs it got every k = 2 test step right but fell short at k = 8.

    Attention does not pass the bit along. At step 9, the model compares step 9 with each of positions 1 to 9 and takes a weighted mix of them. Position 1 is one step away from step 9, however far back it is in the stream. The bottom of the picture shows the two routes.

    The maths

    The stream is x1,…,xTx_1, \dots, x_T with T=12T = 12 (here TT is the number of steps, not a transpose), and the target is yt=xt−ky_t = x_{t-k} for t=k+1,…,Tt = k + 1, \dots, T. For k=8k = 8 that is 4 targets per stream, so the 200 test streams hold 800 steps with a target. The data are made exactly as in recurrent networks, with the same seeds; the recipe is in "About the data" at the end.

  2. Query and keys

    Attention starts with a query and a set of keys, each a vector. Drag the tip of the query q and watch how much weight each key gets.

    The four keys here are placed by hand, in two dimensions. Each key's score is the dot product of the query with the key, divided by dk=2\sqrt{d_k} = \sqrt{2}, where dk=2d_k = 2 is the number of entries in each vector. A key that points the same way as the query gets a high score; one that points away gets a low score (all four keys here have the same length, and a longer key would score higher too). Softmax, from embeddings, turns the four scores into weights between 0 and 1 that add up to 1.

    At the start q = (1.5, 0.5) points closest to k₁: its score is 2.12 and its weight 0.76, and k₂ gets 0.18. To move the query with the keyboard, tab to its tip and use the arrow keys.

    The maths

    For keys k1,…,kn\mathbf{k}_1, \dots, \mathbf{k}_n, the scores and weights are

    si=q⋅kidk,ai=softmax⁡(s)i=exp⁡(si)∑jexp⁡(sj)s_i = \frac{\mathbf{q} \cdot \mathbf{k}_i}{\sqrt{d_k}}, \qquad a_i = \operatorname{softmax}(\mathbf{s})_i = \frac{\exp(s_i)}{\sum_{j} \exp(s_j)}

    so every aia_i is positive and ∑iai=1\sum_i a_i = 1. Here k1=(2,0)\mathbf{k}_1 = (2, 0), k2=(0,2)\mathbf{k}_2 = (0, 2), k3=(−2,0)\mathbf{k}_3 = (-2, 0) and k4=(0,−2)\mathbf{k}_4 = (0, -2).

    Why divide by dk\sqrt{d_k}? Suppose all the 2dk2 d_k entries of q\mathbf{q} and k\mathbf{k} are independent, each with mean 0 and variance 1. Then q⋅k=∑j=1dkqjkj\mathbf{q} \cdot \mathbf{k} = \sum_{j=1}^{d_k} q_j k_j is a sum of dkd_k terms, each with mean 0 and variance 1, so it has mean 0 and variance dkd_k. Dividing by dk\sqrt{d_k} brings the variance back to 1, whatever dkd_k is. Without it, wide vectors give widely spread scores, softmax puts nearly all the weight on one key, and the gradients it passes back become tiny. The independence is a simplification: after training, queries and keys are not independent, but the scaling keeps the scores in a sensible range at the start, when the weights are set up so the entries have roughly unit variance.

  3. A weighted average of values

    Each key comes with a value. The answer is the weighted average of the values, with the weights from the last step.

    The values here are the four corners of a square, also placed by hand: v₁ goes with k₁, and so on. Move q towards k₂ and the output slides towards v₂. Because the weights are positive and add up to 1, the output always lies inside the square. At the start it is (1.22, 1.78): mostly v₁, with some v₂.

    This is why attention is called a soft lookup. A dictionary lookup returns the value stored under the one key that matches exactly. Attention returns a blend of every value, weighted by how well each key matches the query.

    The maths
    o=∑i=1naivi\mathbf{o} = \sum_{i=1}^{n} a_i \mathbf{v}_i

    with v1=(2,2)\mathbf{v}_1 = (2, 2), v2=(−2,2)\mathbf{v}_2 = (-2, 2), v3=(−2,−2)\mathbf{v}_3 = (-2, -2) and v4=(2,−2)\mathbf{v}_4 = (2, -2). With weights that are positive and add up to 1, o\mathbf{o} is a convex combination of the values, so it lies in the smallest convex shape that holds them, here the square. If one weight is near 1, the output is near that one value: a hard lookup is the limit of a soft one.

  4. Queries, keys and values from the input

    In a model, nothing is placed by hand. Each position's input gives its own query, key and value, through weights learned in training.

    Each bit becomes a vector of 16 numbers: a learned vector for the bit (0 or 1) plus a learned vector for its position (1 to 12), called the token and position embeddings. Three weight matrices turn that vector into the position's query, key and value. Position t compares its query with the keys of positions 1 to t. It may not look at later positions, which is called causal attention: those cells are hatched, and their weights are exactly 0. So the map has 78 allowed cells of 144, and each row's weights add up to 1. Row 1 has only one allowed cell, so its weight is 1.

    The attention output of each position is mapped back to 16 numbers, added to the position's own vector (a shortcut), and one more set of weights turns the sum into the probability p^t\hat{p}_t that the target is 1. The model has 1,265 weights and biases. The map shows the starting model, with random weights, on test stream 1: its weights are spread out. Over the 800 rows of the test streams that have a target, the largest weight is 8 positions back in only 21.3%.

    The maths

    With E\mathbf{E} the token embeddings (one row per bit) and P\mathbf{P} the position embeddings (one row per position), position t starts as xt=Ext+Pt\mathbf{x}_t = \mathbf{E}_{x_t} + \mathbf{P}_t, 16 numbers. Then

    qt=WQxt,kt=WKxt,vt=WVxt\mathbf{q}_t = \mathbf{W}_Q \mathbf{x}_t, \qquad \mathbf{k}_t = \mathbf{W}_K \mathbf{x}_t, \qquad \mathbf{v}_t = \mathbf{W}_V \mathbf{x}_t

    each with dk=16d_k = 16 entries. Stacking the rows into matrices Q\mathbf{Q}, K\mathbf{K} and V\mathbf{V} (12 rows each), all positions at once are

    Attention⁡(Q,K,V)=softmax⁡ ⁣(QKTdk)V=AV\operatorname{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \operatorname{softmax}\!\Big(\frac{\mathbf{Q}\mathbf{K}^{\mathsf{T}}}{\sqrt{d_k}}\Big)\mathbf{V} = \mathbf{A}\mathbf{V}

    with the softmax taken along each row. The causal rule: in row t, only the entries s≤ts \le t enter the softmax; the others are left out altogether, so their weights are exactly 0 (not a large negative number that rounds to nearly 0). Then

    zt=wout⋅(xt+WO ot)+b,p^t=σ(zt)z_t = \mathbf{w}_{\text{out}} \cdot \big(\mathbf{x}_t + \mathbf{W}_O\, \mathbf{o}_t\big) + b, \qquad \hat{p}_t = \sigma(z_t)

    where ot\mathbf{o}_t is row t of AV\mathbf{A}\mathbf{V}, and σ here is the sigmoid, not a standard deviation. The count: 2×16+12×16+4×16×16+16+1=12652 \times 16 + 12 \times 16 + 4 \times 16 \times 16 + 16 + 1 = 1265.

  5. Train it (k = 8)

    Press Train. The model is trained to say the bit from 8 steps ago, and the map shows where each position puts its weight.

    Play steps through the lesson, and Train trains the model. Training runs Adam (from optimisers) with η = 0.02 on batches of 10 streams for 10 epochs, 200 batches: the same settings and budget as the recurrent network. The masked test accuracy is the share of the 800 test steps with a target where the predicted bit equals the target. The column on the right of the map is test stream 1's targets, each marked ✓ or ✗.

    These numbers are for the whole run. At the start the masked test accuracy is 49%, about chance. After batch 25 it is 95.5%, and from batch 28 every one of the 800 masked test steps is right, to the end: 100%.

    Watch the map while it trains: a line of dots forms 8 cells left of the diagonal. Position t puts its weight on position t − 8, the very bit it has to say. Over all 800 rows with a target, the largest weight is 8 back in 100% of them, with a mean weight there of 0.97. Rows 1 to 8 have no target, so the loss does not depend on where they look; their pattern is whatever the shared weights happen to give.

    The first two epochs are shown slowly, so the band can be watched forming (with reduced motion it goes straight to the end); the numbers are the same at any speed. Reset goes back to the same starting weights, and the next run is the same as the last.

    The maths

    The loss for one stream is the binary cross-entropy, the log-loss of logistic regression, averaged over the steps that have a target:

    L=1T−k∑t=k+1T(−ytln⁡p^t−(1−yt)ln⁡(1−p^t))\mathcal{L} = \frac{1}{T - k}\sum_{t = k+1}^{T} \Big(-y_t \ln \hat{p}_t - (1 - y_t)\ln(1 - \hat{p}_t)\Big)

    A batch's loss is the mean over its 10 streams. The predicted bit is 1 when p^t>0.5\hat{p}_t > 0.5 and 0 otherwise. With one attention layer and nothing after it but the output weights, the prediction at step t depends only on row t of the map, so the loss does not depend on rows 1 to k at all.

  6. Distance no longer matters

    Move the k slider through 1, 2, 4 and 8. Each of these four k trains a new model from the same starting weights, with the same settings and the same budget of 10 epochs.

    The four runs are computed when you first open this step. All four end at 100% for k = 1, 100% for k = 2, 100% for k = 4 and 100% for k = 8, and every masked test step is right at every epoch from epoch 2 on. For each k, in the rows with a target, the dots sit k cells left of the diagonal.

    What lets position t find the position exactly k back? Not the bits: every 0 starts from the same vector, and so does every 1. It is the learned position embeddings. Training shapes them, together with the query and key weights, so that the query at t scores highest against the key at t − k. Set the trained k = 8 model's position embeddings to zero and its masked test accuracy falls to 53.6%, close to chance.

    The two models also differ in size: the recurrent network had 89 weights and biases, this model 1,265, so this is not a like-for-like contest. What the runs here show is the route. The recurrent network had to pass each bit along, one step at a time; attention reaches any earlier position in one step, so 8 back costs the same as 1 back: k = 1, 2, 4 and 8 all end at 100%, and without position embeddings the trained k = 8 model falls to 53.6%. Attention has its own cost: every position scores every allowed earlier position, so the number of scores grows with the square of the sequence length. Real systems use the same idea at far larger scale.

    The maths

    Without position embeddings, xs=Exs\mathbf{x}_s = \mathbf{E}_{x_s} depends only on the bit. Two earlier positions with the same bit then have the same key and the same value, so they get the same weight and add the same thing to ot\mathbf{o}_t. The output at step t can depend only on the bit at t and on how many 0s and 1s are among positions 1 to t, not on their order, so it cannot say which bit sat at t−kt - k.

    A stream of length TT has T(T+1)/2T(T + 1)/2 allowed pairs under the causal rule, 78 for T=12T = 12, and each pair costs one dot product of length dkd_k. Doubling TT roughly quadruples the work.

Quick check

What do the attention weights in one row add up to?

Right. Softmax divides each exponential by their sum, so the weights are positive and add up to 1. Positions after t are left out, with weight exactly 0.

Each row shares out a total of 1 among its allowed positions. Row 1 has one allowed cell, with weight 1; row 12 has 12, sharing the same total.

Softmax weights are never negative: each is an exponential divided by a sum of exponentials.

How well the query matches changes how the total is shared, not the total. However you drag the query, the four weights still add up to 1.

Why can attention handle k = 8 as easily as k = 2 here?

Right. Position t reaches position t − k in one step, whatever k is. The bits alone cannot say which position is k back: with the position embeddings set to zero, the trained k = 8 model falls to 53.6%.

There is no hidden state carried from step to step. Each position looks back at the earlier positions' keys and values directly.

Every k used the same budget: Adam with η = 0.02, batches of 10 and 10 epochs, from the same starting weights.

k = 1 has the most targets, 11 per stream, and also reached 100%. Fewer targets mean less to learn from, not an easier task.

Lesson complete

  • Attention scores a query against every key, q⋅k/dk\mathbf{q} \cdot \mathbf{k} / \sqrt{d_k}, turns the scores into weights with softmax, and returns the weighted average of the values.
  • In a model, each position's input gives its query, key and value; with the causal rule, position t looks only at positions 1 to t.
  • Within the same 10-epoch budget, this model got every masked test step right for k = 1, 2, 4 and 8, because any earlier position is one step away and the position embeddings let it find exactly k back.

Common misconceptions

  • "An attention weight is the model's confidence." It is the share of the average taken from one position; a row's weights add up to 1 whatever the prediction is.
  • "Attention can see the whole sequence." With the causal rule, position t sees only positions 1 to t.
  • "Attention can tell the order of the inputs by itself." Without position embeddings it cannot tell two positions with the same bit apart; the trained model then fell to 53.6%.
  • "Attention is free." The number of scores grows with the square of the sequence length.
Next lesson: Transformers →

Further reading

  1. Jurafsky & Martin, Speech and Language Processing (3rd ed. draft), Chapter 7, "Transformers and Pretraining".
  2. Bahdanau, Cho & Bengio, "Neural Machine Translation by Jointly Learning to Align and Translate", arXiv 1409.0473 (2014), in which a translation model, at each output word, takes a weighted average of its encodings of every input word.
  3. Vaswani et al., "Attention Is All You Need", arXiv 1706.03762 (2017), which defines the scaled dot-product attention used here.

About the data. Steps 2 and 3 use four keys and four values placed by hand: k₁ = (2, 0), k₂ = (0, 2), k₃ = (−2, 0), k₄ = (0, −2) and v₁ = (2, 2), v₂ = (−2, 2), v₃ = (−2, −2), v₄ = (2, −2); the query starts at (1.5, 0.5) and moves in steps of 0.1. Every stream is 12 bits made up in your browser by the mulberry32 random number generator, exactly as in recurrent networks: for each stream in turn, 12 uniform draws u, and the bit is 1 if u < 0.5 and 0 otherwise. The target at step t is the bit at step t − k, for t = k + 1 to 12. The 200 training streams use seed 211 and the 200 test streams seed 212, for every k. The model's starting weights use seed 271, in this order: the 2 × 16 token embeddings and the 12 × 16 position embeddings, each a normal draw with mean 0 and SD 0.5; the 16 × 16 matrices WQ, WK, WV and WO and the 16 output weights, SD 1/4; the output bias starts at 0. Each epoch's shuffle of the training streams comes from seed 272. Every run uses Adam with η = 0.02, β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸, batches of 10 streams and 10 epochs.

If this helped, you can .