Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 5 · Modern AI

Embeddings: words as points

An embedding is a short list of numbers for each word, learned from text. Train one live and watch words that are used in the same way end up pointing the same way.

About 10 minutes · Every vector, loss and cosine is computed in your browser; the formulas are in The maths

Overview

In tokens, text became a list of ids from a fixed vocabulary. An id is only a label: that "figs" has one number and "plums" another says nothing about either. This lesson gives each word a short list of numbers instead, its embedding, and lets training choose those numbers.

The text is the same invented world as in tokens: eight made-up people, six foods and four places, in short sentences such as "tom lives by the lake.". The world is invented, and its facts are rules of this page, not claims about anyone. This time the text is split into words, not characters.

You'll learn

  • why one-hot vectors cannot say which words are alike
  • how predicting each word's neighbours learns a vector for every word (the skip-gram model)
  • how cosine similarity measures closeness, and what a high cosine does and does not mean
  1. Words as one-hot vectors

    The text has 27 different words, counting the full stop as a word. Number them 0 to 26 and turn each into a one-hot vector: 27 numbers, all 0 except a single 1 in that word's place.

    Every pair of different words is then exactly the same distance apart, √2 = 1.414, and every pair is at right angles. The page checks all 351 pairs. So "mia" is as far from "tom" as from "figs".

    One-hot vectors are fine as labels, but they carry no notion of "similar". For that, words need vectors whose positions are learned from how the words are used.

    The maths

    The vocabulary is V\mathcal{V}, here with ∣V∣=27|\mathcal{V}| = 27 words. Word ii becomes the one-hot vector ei\mathbf{e}_i, with a 1 in place ii. For two different words, ei−ej\mathbf{e}_i - \mathbf{e}_j has one entry 1, one entry −1 and the rest 0, so

    ∥ei−ej∥=12+(−1)2=2,ei⋅ej=0(i≠j)\lVert \mathbf{e}_i - \mathbf{e}_j \rVert = \sqrt{1^2 + (-1)^2} = \sqrt{2}, \qquad \mathbf{e}_i \cdot \mathbf{e}_j = 0 \quad (i \ne j)

    A dot product of 0 means the two vectors are at right angles. Neither number depends on which two words they are.

  2. Learn from neighbours

    The idea: describe a word by the words around it. Each word in a sentence, the centre, makes a pair with every word up to 2 places before or after it in the same sentence.

    Move the centre word slider along "tom lives by the lake.", sentence 2 of the text. The neighbours are dashed. The 1,500 sentences give 22,344 pairs (centre, neighbour) in all.

    The model gives every word 2 numbers, its embedding: the point drawn in the next step. Each word also gets a second pair of numbers, used when it is the neighbour being predicted. Given a centre word, the model scores every word in the list by multiplying the centre's embedding with that word's second pair, and turns the scores into probabilities with softmax. Training adjusts all 108 numbers so that the neighbours that really occur get higher probability. This is the skip-gram model.

    The maths

    The embedding matrix E\mathbf{E} has one row per word, of width d=2d = 2; row cc is wc=ETec\mathbf{w}_c = \mathbf{E}^{\mathsf{T}} \mathbf{e}_c, so multiplying by a one-hot vector picks out one row. The output matrix Wout\mathbf{W}_{\text{out}} has rows uj\mathbf{u}_j. For centre cc, the probability that a neighbour is word oo is

    p(o∣c)=softmax⁡(Woutwc)o=exp⁡(uo⋅wc)∑j∈Vexp⁡(uj⋅wc)p(o \mid c) = \operatorname{softmax}(\mathbf{W}_{\text{out}} \mathbf{w}_c)_o = \frac{\exp(\mathbf{u}_o \cdot \mathbf{w}_c)}{\sum_{j \in \mathcal{V}} \exp(\mathbf{u}_j \cdot \mathbf{w}_c)}

    The loss is the mean, over all NN = 22,344 pairs, of the negative log-probability (natural log):

    L=−1N∑(c, o)ln⁡p(o∣c)\mathcal{L} = -\frac{1}{N} \sum_{(c,\, o)} \ln p(o \mid c)

    A model that gives every word the same probability, 1/271/27, has L=ln⁡27\mathcal{L} = \ln 27 = 3.30. The 108 numbers are 27 × 2 in E\mathbf{E} and 27 × 2 in Wout\mathbf{W}_{\text{out}}.

  3. Press Train

    Press Train. Each word is a point: names are circles, foods triangles, places squares and the other words diamonds. They start as a random scatter.

    Play steps through the lesson, and Train trains the embeddings. Training runs Adam (from optimisers) with η = 0.05 for 200 steps, and every step uses all the pairs at once. The loss ℒ starts at 3.34, a little above the 3.30 of giving every word the same probability, and ends at 2.51. It cannot reach 0: after "tom" comes "likes" in some sentences and "lives" in others, and even in one sentence a word can have up to four neighbours, each a different word, so no model can be sure of a word's neighbours.

    Watch the groups form: names to one side, foods to another, places to a third. Nobody told the model which words are names; it only saw which words sit near which.

    The names end with much shorter vectors than the foods and places, so the picture draws each word in its true direction from the centre, with its length on a square-root scale so that short and long vectors both fit; the dashed rings mark lengths 1 and 4. Direction is what the next step measures. Reset goes back to the same starting numbers, and the next run is the same as the last. The run takes a few seconds on a laptop and longer on a phone.

    The maths

    For one pair, write zj=uj⋅wcz_j = \mathbf{u}_j \cdot \mathbf{w}_c and pj=softmax⁡(z)jp_j = \operatorname{softmax}(\mathbf{z})_j. The gradient of −ln⁡po-\ln p_o is

    ∂(−ln⁡po)∂zj=pj−[j=o],∂(−ln⁡po)∂uj=(pj−[j=o]) wc,∂(−ln⁡po)∂wc=∑j(pj−[j=o]) uj\frac{\partial (-\ln p_o)}{\partial z_j} = p_j - [j = o], \qquad \frac{\partial (-\ln p_o)}{\partial \mathbf{u}_j} = (p_j - [j = o])\, \mathbf{w}_c, \qquad \frac{\partial (-\ln p_o)}{\partial \mathbf{w}_c} = \sum_{j} (p_j - [j = o])\, \mathbf{u}_j

    where [j=o][j = o] is 1 when j=oj = o and 0 otherwise. Each step averages these over all 22,344 pairs (full-batch, not mini-batches) and Adam moves every number, with β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999 and ε=10−8\varepsilon = 10^{-8}. The drawn point for a vector of length rr is at distance r\sqrt{r} from the centre, in the same direction.

  4. Used alike means close

    Cosine similarity is the cosine of the angle between two vectors: 1 when they point the same way, 0 at right angles, −1 when opposite. Pick a word to see its five nearest words by cosine. Judge closeness by the angle of the rays from the centre, not by the gap between the points.

    After the run, the nearest word to every name is a name (8 of 8), to every food a food (6 of 6) and to every place a place (4 of 4). At the random start it was 2 of 8, 0 of 6 and 0 of 4. If you have not trained yet, press Train: the table always shows the model as it is now.

    Close does not mean "seen together". No sentence holds two different foods (the page counts 0), yet after the run "apples" and "pears" have cosine 1.000 to three decimal places. They are close because they share neighbours: a name two words before them, "likes" just before and the full stop after.

    The maths

    For vectors a\mathbf{a} and b\mathbf{b} at angle θ\theta,

    cos⁡θ=a⋅b∥a∥ ∥b∥\cos\theta = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert\, \lVert \mathbf{b} \rVert}

    Cosine ignores length: doubling a vector does not change it. The table shows the angle too, in degrees, because many cosines here round to 1.000 while their angles still differ.

  5. Search by meaning

    Pick a word, and every other word is ranked by cosine with it. In its simplest form, search by meaning works this way: turn the query into a vector and return the items whose vectors point most nearly the same way.

    Averaged over every pair of different words in a group, the cosine after the run is 0.95 for names, 1.00 for foods and 1.00 for places; at the random start it was −0.05, −0.10 and 0.18. Between groups it is below 0: −0.89 for a name and a place, −0.22 for a name and a food and −0.21 for a food and a place.

    A search for "figs" puts the 5 other foods first, but "are", "and" and "friends" come straight after, each with cosine 1.000 with every food to three decimal places; "the" has cosine 1.000 with every place. These words share only some of their neighbours with the foods or places ("friends", like a food, has a name two words before it and the full stop after). With only two numbers per word there are few directions to go round, so words that are only partly alike end up pointing the same way too. Real embeddings have hundreds or thousands of numbers per word and far more room.

    "Meaning" here is only "used in similar places in the text", and even that only roughly. The smallest cosine between two foods is 1.000 to three decimal places, so a search for "figs" cannot tell the foods apart, even though the text says who likes which.

    The maths

    For groups AA and BB, the mean cosine is taken over every pair of different words a∈Aa \in A, b∈Bb \in B:

    cos⁡‾(A,B)=1#{(a,b):a≠b}∑a≠bcos⁡(wa,wb)\overline{\cos}(A, B) = \frac{1}{\#\{(a, b) : a \ne b\}} \sum_{a \ne b} \cos(\mathbf{w}_a, \mathbf{w}_b)

    The ranking uses the embeddings w\mathbf{w}, the rows of E\mathbf{E}, not the output vectors u\mathbf{u}.

  6. Real embeddings

    Real systems use the same idea at far larger scale: hundreds or thousands of numbers per word or token, learned from huge amounts of text, and compared by cosine to search by meaning.

    Ours has 2 numbers per word, 27 × 2 in all, so it can be drawn on a page. In two dimensions a word's direction is a single angle around a circle, so there is little room: the six foods, which differ only in who likes them, share one direction, and "are", "and" and "friends" share it with them. More numbers per word leave room to separate far more kinds of use.

    The skip-gram model was introduced by Mikolov, Chen, Corrado and Dean in 2013. They trained it on news text of up to about 6 billion words, and to cope with a vocabulary of a million words they replaced the full softmax used here with a cheaper version (hierarchical softmax). In the transformers later in this level, the embeddings are a first layer learned together with the rest of the model, but the idea is the same: a vector per token, learned from text. Like ours, real embeddings reflect how words are used in their training text, including patterns nobody intended.

    The maths

    The two axes of the picture have no fixed meaning. Rotating every wc\mathbf{w}_c and every uj\mathbf{u}_j by the same rotation R\mathbf{R} leaves every dot product, so every probability, unchanged, because (Ru)⋅(Rw)=u⋅w(\mathbf{R}\mathbf{u}) \cdot (\mathbf{R}\mathbf{w}) = \mathbf{u} \cdot \mathbf{w}. So the axes mean nothing on their own; the angles between the vectors are what the picture can show.

Quick check

Why are one-hot vectors no good for similarity?

Right. All 351 pairs of different words are 1.414 apart with a dot product of 0, whichever words they are.

They have 27 numbers each, more than the 2 of the embeddings here. The trouble is not length: every pair is equally far apart.

Nothing is random: each word's vector has its single 1 in the word's own place in the list. The embeddings are the ones that start random, and training moves them.

The distance is √2 for every pair of different words, neighbours in the list or not, so the order of the list makes no difference.

Two words have embeddings with cosine near 1. What does that say?

Right, as a rule of thumb. Training tends to pull the vectors of words with the same neighbours towards the same direction. It is a tendency, not a guarantee: in our 2-D model "and", "are" and "friends" share only some neighbours with the foods, yet also have cosine 1.000 with them.

"apples" and "pears" are different foods, yet after the run their cosine is 1.000. They are used in the same way, which is not the same as meaning the same thing.

No sentence holds two different foods, yet the foods end up with cosines near 1. Sharing neighbours is what counts, not appearing together.

Cosine ignores length: doubling a vector does not change its cosine with any other vector.

Lesson complete

  • One-hot vectors put every pair of different words √2 apart, so they cannot show similarity.
  • The skip-gram model learns an embedding per word by predicting each word's neighbours. Here, from 2 numbers per word, the nearest word to every name became a name, to every food a food and to every place a place.
  • Cosine similarity compares directions. Training puts words with similar neighbours in similar directions, so a high cosine suggests two words are used alike, not that they mean the same thing; with few numbers per word, words that are only partly alike can share a direction too.

Common misconceptions

  • "Words with close embeddings mean the same thing." They share neighbours in the training text; here all six foods point almost the same way.
  • "Close words appear together." No sentence here holds two foods, yet the mean cosine between two foods is 1.00.
  • "The model was told which words are names, foods and places." It saw only pairs of nearby words. The groups come from that.
  • "Each axis of an embedding stands for something, such as food or place." Rotating all the vectors together changes nothing the model computes, so the axes have no fixed meaning.
Next lesson: Predicting the next token →

Further reading

  1. Mikolov, Chen, Corrado & Dean (2013), "Efficient Estimation of Word Representations in Vector Space", arXiv:1301.3781, the paper that introduced the skip-gram model.
  2. Jurafsky & Martin, Speech and Language Processing (3rd ed. draft, free online), the chapter "Embeddings", on vector semantics, cosine similarity and the skip-gram model.

About the data. The text is 1,500 sentences of an invented world, made up in your browser by the mulberry32 random number generator with seed 251: for each sentence one uniform draw picks the kind (below 0.4 "<name> likes <food>.", below 0.8 "<name> lives by/in the <place>.", otherwise "<name> and <name> are friends." with two different names) and further draws pick the names uniformly; each person's food and place come from a fixed table. The world is invented and its facts are rules of this page, not claims about anyone. Words are split at spaces, with the full stop as a word of its own; pairs are taken within 2 words, inside one sentence. The starting embeddings and output vectors use seed 252, each number 0.5 times a standard normal draw. The order in which each full batch is summed comes from a shuffle with seed 253; it changes nothing but the last digits.

If this helped, you can .