The Least-Squares Line: Quick Revision

A brief note on The Least-Squares Line. Read it before a test. Try each question at the end before you open its answer.

Residuals

• Residual: $y - \hat{y}$, observed minus predicted. Above the line it is positive.
• SSE: the sum of the squared residuals, the score of a line.
• Least squares: the one line with the smallest SSE. Its residuals add up to $0$.

The line

• Slope: $b = r\,\dfrac{s_y}{s_x} = \dfrac{S_{xy}}{S_{xx}}$, in units of $y$ per unit of $x$.
• Intercept: $a = \bar{y} - b\bar{x}$. The line passes through $(\bar{x}, \bar{y})$.
• Prediction: $\hat{y} = a + bx$. Inside the data is interpolation. Outside is extrapolation, and risky.
• Direction: the line predicts $y$ from $x$. Predicting $x$ from $y$ needs another line.

Checking the line

• Residual plot: a random band means a straight line fits. A curve or a fan means it does not.
• $r^2$: $1 - \dfrac{\text{SSE}}{S_{yy}}$, the share of the variation in $y$ accounted for.
• Back to $r$: $r = \pm\sqrt{r^2}$, with the sign of the slope.

A worked example

Five made-up students revised for $1$ to $5$ hours and scored $52$, $55$, $61$, $62$ and $70$ marks.

1. Means and sums. $\bar{x} = 3$ hours, $\bar{y} = 60$ marks, $S_{xx} = 10$ and $S_{xy} = 43$.
2. The line. $b = 4.3$ marks per hour and $a = 60 - 4.3 \times 3 = 47.1$ marks, so $\hat{y} = 47.1 + 4.3x$.
3. Prediction. For $3.5$ hours, $\hat{y} = 47.1 + 4.3 \times 3.5 = 62.15$ marks. And $r^2 = 0.9531$.

Practice with new numbers

A program to try

The program finds the line, the prediction and $r^2$ of the worked example.

Mistakes to avoid

• Residual the wrong way round. Observed minus predicted.
• The ratio upside down. $s_y$ goes on top in the slope.
• Reading $r$ as the share explained. The share is $r^2$.
• Trusting a high $r$ without the residual plot.

Check yourself

1. A line predicts $25$ and the observed value is $21$. What is the residual?

2. For some data $r = 0.5$, $s_x = 2$, $s_y = 8$, $\bar{x} = 4$ and $\bar{y} = 20$. Find the least-squares line.

3. A line has $r = -0.9$. What share of the variation in $y$ does it account for?

4. The residuals of a line are positive at both ends and negative in the middle. What does that mean?

Sources

• OpenStax, Introductory Statistics, Chapter 12 (Linear Regression and Correlation).