The two regression lines

What you will be able to do

• find the least squares line of $y$ on $x$ and the line of $x$ on $y$;
• use the slopes to find $r$, and use $r$ to find a slope;
• find the means, $r$ and the ratio of the spreads from two given regression lines.

The line of best fit

The marks of five students rise with their study hours. We want one straight line that describes the pattern and lets us predict a mark from the hours. Many lines look reasonable. Which one is best?

For a line $\hat{y} = a + b x$, the residual of a point is the vertical miss $y - \hat{y}$. The least squares line makes $\sum (y - \hat{y})^2$ as small as possible. Using the three sums of the last lesson, it has

Equivalently, the line passes through $(\bar{x}, \bar{y})$ and has the equation

The slope is written $b_{yx}$ and called the regression coefficient of $y$ on $x$. For the five students, $b_{yx} = \dfrac{50}{10} = 5$ and $a = 60 - 5 \times 3 = 45$. So $\hat{y} = 45 + 5x$: each extra hour of study goes with $5$ more marks. Notations differ: A-Level and AP write $\hat{y} = a + bx$, and IB writes $y = ax + b$. The line is the same.

The other line

We can also ask for a line that predicts $x$ from $y$. Now the misses are measured horizontally, and the line is

For the five students, $b_{xy} = \dfrac{50}{256} = 0.195$. The line is $x = -8.72 + 0.195\,y$.

The two lines are not the same line written differently. They both pass through $(\bar{x}, \bar{y})$, and they cross there. The nearer the points are to a line, the closer the two lines are to each other. If $r = \pm 1$ they coincide. If $r = 0$, the line of $y$ on $x$ is horizontal and the line of $x$ on $y$ is vertical.

Facts that link the two lines

• $b_{yx} \times b_{xy} = r^2$, so $r = \pm\sqrt{b_{yx}\,b_{xy}}$, with the sign of the two slopes. Both slopes have the sign of $r$.
• So $b_{yx}\,b_{xy} \le 1$. If a pair of lines gives a product above $1$, the lines were assigned the wrong way round.
• Use the line of $y$ on $x$ to predict $y$ from $x$. Use the line of $x$ on $y$ to predict $x$ from $y$. Do not rearrange one into the other.

A worked example from two lines

The regression lines of a data set are $4x - 5y + 60 = 0$ and $20x - 9y - 20 = 0$. Find the means, which line is which, and $r$.

1. The means. Both lines pass through the point $(\bar{x}, \bar{y})$. Solving them, $\bar{x} = 10$ and $\bar{y} = 20$.
2. Try the first as $y$ on $x$. Its slope is $b_{yx} = \dfrac{4}{5} = 0.8$. The second then is $x$ on $y$: $x = \dfrac{9}{20}\,y + 1$, so $b_{xy} = 0.45$.
3. The test. $0.8 \times 0.45 = 0.36 \le 1$, so this assignment is possible. The other way gives $b_{yx} = \dfrac{20}{9}$ and $b_{xy} = \dfrac{5}{4}$, with product $2.78 > 1$, which is impossible.
4. The coefficient. $r = +\sqrt{0.36} = 0.6$, positive because both slopes are positive.
5. The spreads. $\dfrac{\sigma_y}{\sigma_x} = \dfrac{b_{yx}}{r} = \dfrac{0.8}{0.6} = \dfrac{4}{3}$.

Practise with new numbers

Try it in Python

The program finds both regression lines for the hours and the marks, checks that the product of the slopes is $r^2$, and predicts the mark for $3.5$ hours.

Common mistakes

• Using the wrong line to predict. To predict $x$ from $y$, use the line of $x$ on $y$.
• Turning the line of $y$ on $x$ round to get the other line. The slope of the other line is $\dfrac{S_{xy}}{S_{yy}}$, not $\dfrac{1}{b_{yx}}$.
• Forgetting the sign. $r$ has the sign of $b_{yx}$ and $b_{xy}$.
• Taking the root of a product above $1$. Then the assignment of the lines is wrong.
• Forgetting that the lines meet at the means.

Remember these

• $b_{yx} = \dfrac{S_{xy}}{S_{xx}} = r\dfrac{\sigma_y}{\sigma_x}$, $b_{xy} = \dfrac{S_{xy}}{S_{yy}} = r\dfrac{\sigma_x}{\sigma_y}$, and $b_{yx}\,b_{xy} = r^2$.
• Both lines pass through $(\bar{x}, \bar{y})$.
• The least squares line makes the sum of the squared vertical residuals smallest.
• $y - \bar{y} = b_{yx}(x - \bar{x})$ and $x - \bar{x} = b_{xy}(y - \bar{y})$.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. Data have $\bar{x} = 4$, $\bar{y} = 10$, $r = 0.8$, $\sigma_x = 2$ and $\sigma_y = 5$. Find the line of $y$ on $x$.

2. For the same data, find the line of $x$ on $y$.

3. The two regression coefficients of a data set are $1.2$ and $0.3$. Find $r$.

4. A student says that the two regression coefficients of a data set are $1.5$ and $0.9$. Is this possible?

5. Why does the line of $y$ on $x$ always pass through $(\bar{x}, \bar{y})$?

References

• OpenStax, Introductory Statistics, Chapter 12, sections on the regression equation and the least squares line.
• A-Level, IB and AP texts all give $b = \dfrac{S_{xy}}{S_{xx}}$ for the regression of $y$ on $x$. The regression of $x$ on $y$ and the product $b_{yx}\,b_{xy} = r^2$ are standard in Indian texts.