The line that fits best

What you will be able to do

• find the residual of a point from a line, with its sign;
• explain why we square the residuals before adding them;
• find the sum of squared residuals of a line, and know that the least-squares line makes it smallest.

Drinks on a hot day

A made-up school canteen notes the noon temperature and the cold drinks it sells on six days.

The scatter plot is a strong, positive, straight-line pattern, with $r = 0.987$. The cook would like a rule: "on a day of $x$ degrees, order about $\hat{y}$ drinks". A straight line gives such a rule. We write it as $\hat{y} = a + bx$. The hat on $\hat{y}$ says that it is a predicted value, not an observed one.

You could draw a line by eye. Two people would draw two different lines. We want one line that everybody agrees on.

The miss of each point

For each day, compare what happened with what the line predicts. The residual is

Take the line $\hat{y} = 2x$, a first guess. On the day of $24\ ^\circ\text{C}$ it predicts $48$ drinks, and the canteen sold $47$. The residual is $47 - 48 = -1$ drink: the point is $1$ drink below the line. A point above the line has a positive residual.

Why square them?

The natural idea is to add the residuals and make the sum small. It does not work. Positive and negative residuals cancel. In fact every line through the point of the means $(\bar{x}, \bar{y})$ has residuals that add up to exactly $0$, however badly it fits.

So we square each residual first. A square is never negative, so misses cannot cancel. Squares also punish a big miss more than two small ones. The sum of squared residuals (SSE, for sum of squared errors) is the score of a line: the smaller, the better.

The least-squares line is the one line with the smallest possible SSE. For the canteen it is $\hat{y} = -9.13 + 2.432x$, with SSE $= 20.72$. No other straight line does better. The first guess $\hat{y} = 2x$ has SSE $= 62$.

One more point: the least-squares line measures the misses up and down, along the $y$ direction, because $y$ is the quantity we want to predict.

Your turn: new numbers

Try it in Python

The program scores two lines drawn by eye, and then the least-squares line that the statistics module finds.

The usual mistakes

• Getting the sign the wrong way round. The residual is observed minus predicted. A point above the line has a positive residual.
• Adding the residuals without squaring. They cancel, and every line through the means gives $0$.
• Measuring the misses across, or at right angles to the line. Least squares measures them up and down.
• Thinking the best line passes through the most points. It may pass through none of them.

Remember these

• The residual is $y - \hat{y}$, observed minus predicted.
• SSE is the sum of the squared residuals, the score of a line.
• The least-squares line has the smallest SSE of all straight lines.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. A line predicts $43.5$ drinks on a day when the canteen sold $40$. What is the residual?

2. A friend says the best line is the one whose residuals add up to zero. Why is that not enough?

3. The line $\hat{y} = 1 + 2x$ is drawn through the points $(1, 4)$, $(2, 4)$ and $(3, 8)$. Find the residuals and the SSE.

4. When is the SSE of a line equal to $0$?

5. Why are the misses measured up and down, and not along the $x$ direction?

References

• OpenStax, Introductory Statistics, Chapter 12 (Linear Regression and Correlation), the section on the regression equation.