Variance and standard deviation

What you will be able to do

• find the variance and the standard deviation of a list, a frequency table and grouped data;
• use the shortcut and the step-deviation methods;
• explain the difference between dividing by $n$ and by $n - 1$.

Squaring instead of dropping the sign

The mean deviation removes the sign by taking the absolute value. There is another way to remove it: square each deviation. Squares are always positive, and they make a large miss count much more than a small one. Take the mean of the squared deviations and you have the variance, written $\sigma^2$ (sigma squared):

With frequencies, the variance is

The variance is in squared units, so we take the square root and get the standard deviation:

It has the same unit as the data. For machine A of the last lesson, the squared distances from $500$ are $4, 1, 0, 1, 4$, which add to $10$. So $\sigma^2 = \dfrac{10}{5} = 2\ \text{g}^2$ and $\sigma = 1.41\ \text{g}$. For machine B, the squared distances are $400, 100, 0, 100, 400$, so $\sigma^2 = 200\ \text{g}^2$ and $\sigma = 14.14\ \text{g}$. The ratio of the standard deviations is $10$, the same as for the mean deviations.

The shortcut method

The direct method needs the mean first, and then every distance. Expand the bracket:

Divide by $n$:

In words, the variance is the mean of the squares minus the square of the mean. For a frequency table, use $\dfrac{\sum f x^2}{N} - \bar{x}^2$. This needs only the two totals $\sum f x$ and $\sum f x^2$.

The step-deviation method

For grouped data with class marks far from zero, write $u = \dfrac{x - A}{h}$ with an assumed mean $A$ and class width $h$. Shifting by $A$ does not change a spread, and dividing by $h$ divides the spread by $h$. So

The simulation shows why the mean is the natural centre for the variance. For any centre $a$,

The first term is the variance. The second term is never negative, and it is zero only when $a = \bar{x}$. So the mean square deviation is smallest about the mean. This identity returns when we combine two groups in the next course.

A worked example with grouped data

Use the marks of $50$ students once more: class marks $5, 15, 25, 35, 45, 55$, frequencies $4, 8, 15, 10, 7, 6$, and mean $30.2$.

1. Shortcut totals. $\sum f x^2 = 100 + 1800 + 9375 + 12250 + 14175 + 18150 = 55850$.
2. Mean of the squares. $\dfrac{55850}{50} = 1117$.
3. Subtract the square of the mean. $30.2^2 = 912.04$, so $\sigma^2 = 1117 - 912.04 = 204.96$.
4. The standard deviation. $\sigma = \sqrt{204.96} = 14.32$ marks.

Now the step-deviation check, with $A = 25$ and $h = 10$. The values of $u$ are $-2, -1, 0, 1, 2, 3$. Then $\sum f u = 26$ and $\sum f u^2 = 116$, so $\sigma^2 = 100\left[\dfrac{116}{50} - \left(\dfrac{26}{50}\right)^2\right] = 100\,(2.32 - 0.2704) = 204.96$. The two methods agree.

Same mean, different spread

Two classes both have a mean mark of $60$. The marks of class A lie close to $60$, and the marks of class B are spread from $20$ to $100$. The standard deviation tells them apart, as the mean cannot. Class A has $\sigma = 5.2$ marks and class B has $\sigma = 25.8$ marks.

Population and sample

Everything above treats the data as the whole group, and divides by $n$. This is what the Indian syllabus does. Often, however, the data are only a sample taken from a much larger population, and we want to estimate the population variance from them. A sample tends to be closer to its own mean than to the population mean, so dividing by $n$ makes the estimate a little too small. Dividing by $n - 1$ corrects this:

A-Level, IB and AP Statistics write $s$ for this, call it the sample standard deviation (and sometimes the unbiased estimate of the variance), and use it when the data are a sample. Calculators show $\sigma_n$ and $s_{n-1}$ as two separate keys. For a large $n$ the two are almost equal. If a question says "the variance of the data" or "of these $n$ numbers", divide by $n$. If it says "an estimate of the population variance" or "the sample variance", divide by $n - 1$.

Practise with new numbers

Try it in Python

The program finds the variance of the grouped marks by the direct method, the shortcut and the step-deviation method. All three should give the same number.

Common mistakes

• Forgetting the square root. The variance is in squared units. The standard deviation is its square root.
• Subtracting the mean instead of its square. The shortcut is $\dfrac{\sum f x^2}{N} - \bar{x}^2$.
• Forgetting $h^2$. In the step-deviation method the answer is multiplied by $h^2$ for the variance, and by $h$ for the standard deviation.
• Using $n - 1$ when the whole group is given. Divide by $n$ unless the question says sample.

Remember these

• $\sigma^2 = \dfrac{\sum f (x - \bar{x})^2}{N} = \dfrac{\sum f x^2}{N} - \bar{x}^2$ and $\sigma = \sqrt{\sigma^2}$.
• Step deviation: $\sigma^2 = h^2\left[\dfrac{\sum f u^2}{N} - \left(\dfrac{\sum f u}{N}\right)^2\right]$.
• The variance is never negative, and it is zero only when all the values are equal.
• Divide by $n - 1$ for a sample, as A-Level, IB and AP do.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. Find the variance and the standard deviation of $2, 4, 6, 8, 10$.

2. For $10$ values, $\sum x = 60$ and $\sum x^2 = 400$. Find the variance and the standard deviation.

3. Can a variance be negative? When is it exactly zero?

4. In a step-deviation calculation, $A = 40$, $h = 10$, $N = 30$, $\sum f u = 6$ and $\sum f u^2 = 48$. Find the mean and the variance.

5. The values are $2, 4, 4, 6$. Find the variance of the data as a whole group, and the sample variance.

References

• OpenStax, Introductory Statistics, Chapter 2, section on measures of the spread of the data (variance and standard deviation).
• NCERT, Mathematics for Class XI, Chapter 13 (Statistics), section on variance and standard deviation.