Correlation and Regression: Quick Revision

A brief note on Correlation and Regression. Read it before a test. Try each question at the end before you open its answer.

Correlation

• Sums: $S_{xx} = \sum (x - \bar{x})^2$, $S_{yy} = \sum (y - \bar{y})^2$, $S_{xy} = \sum (x - \bar{x})(y - \bar{y})$.
• Coefficient: $r = \dfrac{S_{xy}}{\sqrt{S_{xx}\,S_{yy}}}$, with $-1 \le r \le 1$ and no unit.
• Meaning: $r$ measures straight-line association only. $r = 0$ does not mean no relationship.
• Changes: adding a number or multiplying by a positive number leaves $r$ unchanged. A negative multiplier flips its sign.
• Rank correlation: $r_s = 1 - \dfrac{6\sum d^2}{n(n^2 - 1)}$.

Regression

• $y$ on $x$: $y - \bar{y} = b_{yx}(x - \bar{x})$, with $b_{yx} = \dfrac{S_{xy}}{S_{xx}} = r\dfrac{\sigma_y}{\sigma_x}$.
• $x$ on $y$: $x - \bar{x} = b_{xy}(y - \bar{y})$, with $b_{xy} = \dfrac{S_{xy}}{S_{yy}} = r\dfrac{\sigma_x}{\sigma_y}$.
• Link: $b_{yx}\,b_{xy} = r^2$. Both lines pass through $(\bar{x}, \bar{y})$.
• Least squares: the sum of the squared vertical misses is smallest.

Prediction and traps

• Interpolation is safe. Extrapolation is risky.
• $r^2$ is the share of the variation in $y$ that the line explains.
• One odd point can change $r$ and the line a lot.
• Correlation is not cause. Look for a lurking variable.
• Mixed groups can reverse a pattern.

A worked example

Study hours $1, 2, 3, 4, 5$ and marks $50, 56, 58, 66, 70$.

1. The sums. $\bar{x} = 3$, $\bar{y} = 60$, $S_{xx} = 10$, $S_{yy} = 256$ and $S_{xy} = 50$.
2. The correlation. $r = \dfrac{50}{\sqrt{2560}} = 0.988$.
3. The line of $y$ on $x$. $b = \dfrac{50}{10} = 5$ and $a = 60 - 15 = 45$, so $\hat{y} = 45 + 5x$.
4. A prediction. At $3.5$ hours, $\hat{y} = 62.5$. At $12$ hours it gives $105$, which is extrapolation and cannot be right.

Practice with new numbers

A program to try

The program finds $r$, the line of best fit and a prediction for a short data set.

Mistakes to avoid

• Reading $r = 0$ as no relationship.
• Reading a strong $r$ as proof of cause.
• Using the line of $y$ on $x$ to predict $x$.
• Extrapolating far outside the data.
• Reading $r$ as the share explained. It is $r^2$.

Check yourself

1. Find $r$ when $S_{xx} = 9$, $S_{yy} = 16$ and $S_{xy} = 6$.

2. The regression coefficients are $0.8$ and $0.45$. Find $r$.

3. The line of $y$ on $x$ is $y = 3 + 2x$ and $\bar{x} = 5$. Find $\bar{y}$.

4. For $r = -0.7$, find the share of the variation in $y$ explained by the line.

5. A line was fitted to data with $x$ from $5$ to $20$. Is it sensible to predict $y$ at $x = 80$?

Sources

• OpenStax, Introductory Statistics, Chapter 12 (Linear Regression and Correlation).
• A-Level, IB and AP Statistics texts on the product moment correlation coefficient and least squares regression.