Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 0 · Start here

What is a model?

A model is a rule learned from examples. Once it has learned, it can make a guess about a case it has never seen.

About 8 minutes · No maths needed; the maths is there if you want it

Overview

Suppose you have the records of 20 students: how many hours each one studied for an exam, and the mark each one got. A new student tells you they studied for six hours. What mark would you guess?

You would probably look at the students who studied about that long and guess something close to their marks. A machine learning model does the same job, but it writes the idea down as a rule with a few numbers in it. Those numbers are chosen by looking at the examples. Choosing them is called training.

The students in this lesson are made up, so we know exactly how their marks were produced. See "About the data" at the end.

You'll learn

  • what a model is, and what its parameters are
  • what it means to train a model on examples
  • why a good model's predictions are still not exact
  1. Start with examples

    Each dot is one student. Across is the hours they studied; up is the mark they got, out of 100.

    These 20 dots are the examples the model will learn from. Each example has an input, the hours, and a known answer, the mark.

    Look at the overall shape. Students who studied longer tend to have higher marks, but the dots do not sit on a neat line.

    The maths

    We write the examples as pairs (xi,yi)(x_i, y_i) for i=1,…,ni = 1, \dots, n. Here xix_i is student ii's hours, yiy_i is their mark, and n=20n = 20.

    The input xx is also called a feature, and the answer yy the target.

  2. A rule is a guess

    A model is a rule that turns hours into a guessed mark. This rule is a straight line: start at a, then add b marks for every hour studied.

    The line is the rule. The mark it guesses is written ŷ, said "y-hat". Move the sliders: a sets where the line starts at zero hours, and b sets how steep it is.

    The short vertical lines show how far each real mark is from the guess. The "average squared miss" on the stage multiplies each of those gaps by itself and takes the average. Smaller is better. Lesson 4 explains why we square.

    The starting line is deliberately poor: it says that more study means a lower mark.

    The maths
    y^=f(x; θ)=a+b x,θ=(a,b)\hat{y} = f(x;\,\boldsymbol{\theta}) = a + b\,x, \qquad \boldsymbol{\theta} = (a, b)

    The numbers aa (the intercept) and bb (the slope) are the rule's parameters, collected in θ\boldsymbol{\theta} (theta). The semicolon separates the input from the parameters.

    The average squared miss is the mean squared error:

    MSE=1n∑i=1n(yi−y^i)2\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^{n} \left(y_i - \hat{y}_i\right)^2
  3. Training picks the best rule

    Learning means choosing a and b so that the line fits the examples as well as possible.

    The line has moved from your guess to the best fit. Training looks for the a and b that make the average squared miss as small as it can be. For a straight line there is exactly one best answer (as long as the hours are not all the same), and it can be worked out with a formula.

    The dashed line is your guess from the last step, kept for comparison.

    "Best" here means the smallest average squared miss. Other ways of measuring a miss can give different lines.

    The maths

    Training solves

    θ^=(a^,b^)=arg⁡min⁡a, b 1n∑i=1n(yi−a−b xi)2\hat{\boldsymbol{\theta}} = (\hat{a}, \hat{b}) = \arg\min_{a,\,b}\ \frac{1}{n}\sum_{i=1}^{n} \left(y_i - a - b\,x_i\right)^2

    This is least squares. For one input it has a closed-form answer, where xˉ\bar{x} and yˉ\bar{y} are the means of the hours and the marks:

    b^=∑i(xi−xˉ)(yi−yˉ)∑i(xi−xˉ)2,a^=yˉ−b^ xˉ\hat{b} = \frac{\sum_{i}(x_i - \bar{x})(y_i - \bar{y})}{\sum_{i}(x_i - \bar{x})^2}, \qquad \hat{a} = \bar{y} - \hat{b}\,\bar{x}

    It needs the hours to vary: if every student studied the same time, the formula would divide zero by zero, and many lines would tie for best. A hat marks an estimate: a^\hat{a} and b^\hat{b} are learned from data, and y^\hat{y} is the rule's guess. Lesson 6 derives the formula.

  4. Predict a new case

    Now use the rule. Drag the ring, or move the hours slider, to choose a new student's study time and read off the predicted mark.

    The new student is not one of the 20 examples. The model has never seen their mark, and it does not need to: it puts their hours into the rule and reads the guess off the line. This is prediction.

    Training happened once, using the examples' hours and marks. Prediction can happen as often as you like, and needs only the hours.

    The slider stops at 1 and 10 hours, about the range the examples cover. Outside that range the line still gives an answer, but no data backs it up.

    The maths
    y^new=f(xnew; θ^)=a^+b^ xnew\hat{y}_{\text{new}} = f(x_{\text{new}};\,\hat{\boldsymbol{\theta}}) = \hat{a} + \hat{b}\,x_{\text{new}}

    Training uses inputs and answers, (xi,yi)(x_i, y_i), to fix θ^\hat{\boldsymbol{\theta}}. Prediction uses θ^\hat{\boldsymbol{\theta}} and a new input only.

  5. Useful, not perfect

    The line gives a sensible guess, but real marks scatter around it. Some students did better than the rule predicts, and some did worse.

    The vertical lines are the misses left after training. In real data, misses come from everything the rule leaves out: how well someone slept, what they already knew, luck on the day. In these made-up data, the random noise we added plays that part (see About the data).

    A model simplifies on purpose. It keeps the main pattern and ignores the rest. The fitted line is itself an estimate, so it differs a little from the rule used to make the data.

    So a prediction for a new student is a best guess, not a promise.

    One caution: a pattern in data does not, on its own, show cause and effect.

    The maths

    Each miss is a residual, ei=yi−y^ie_i = y_i - \hat{y}_i. A common way to write the whole story is

    y=f(x; θ)+εy = f(x;\,\boldsymbol{\theta}) + \varepsilon

    where ε\varepsilon (epsilon), often called noise, stands for everything the rule does not capture.

Quick check

What does training produce?

Right. Training looks at the examples and settles on the numbers in the rule. After that, the rule is all you need to make a prediction.

Training gives a rule that makes a best guess. The guess is not exact, because marks also depend on things the rule leaves out.

Training uses the examples you already have; it does not create new ones. What it produces is the pair of numbers a and b.

A fitted line shows that hours and marks go together in these examples. That alone does not show that one causes the other.

Why is the prediction for a new student not exact?

Rounding errors in a calculation like this are far too small to matter. The misses you saw are several marks wide.

Right. In real life, sleep, prior knowledge and luck on the day all affect a mark, and the rule only knows the hours. In this lesson's made-up data, random noise plays that part. Even the best line leaves misses.

For a straight line, least squares finds the best a and b exactly. No amount of extra training removes the scatter in the data.

Predicting cases it has not seen is the whole point of a model. It gives a best guess; it just cannot be exact.

Lesson complete

  • A model is a rule with some numbers left open. Those numbers are its parameters.
  • Training chooses the parameters that make the rule fit the examples best.
  • Prediction runs the trained rule on a new input. The answer is useful, not exact.

Common misconceptions

  • "The model stores the examples." This one keeps only two numbers, a and b. The 20 students are not inside it.
  • "A good fit proves cause and effect." It shows that two things go together in the data, nothing more.
  • "A trained model can be trusted for any input." Far outside the range of its examples, it has no data to go on.
  • "If the predictions miss, the model is broken." Some miss is expected, because the rule leaves out things that matter.
Next lesson: Data as numbers →

Further reading

  1. James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), Section 2.1, "What Is Statistical Learning?", and Section 3.1, "Simple Linear Regression".
  2. Deisenroth, Faisal & Ong, Mathematics for Machine Learning, Chapter 8, "When Models Meet Data".

About the data. The 20 students are made up. They are generated in your browser with the mulberry32 random number generator, seed 7. For each student in turn, the hours x are drawn uniformly between 1 and 10, and then the mark is y = 35 + 5x + 6z, where z is a standard normal random number. Marks are clipped to the range 0 to 100. The same seed gives the same students every time.

If this helped, you can .