Project: how sure is a measurement?

What you will be able to do

After this lesson you can

• find the uncertainty of a mean with the standard error,
• simulate a whole experiment many times to see how much its answer varies,
• estimate an uncertainty with the bootstrap, and
• tell a random error from a systematic one.

The problem

You measure $g$, the acceleration of gravity, with a pendulum. A pendulum of length $L$ swings with the period $T = 2\pi\sqrt{L/g}$. So $T^2 = \dfrac{4\pi^2}{g}L$: if you plot $T^2$ against $L$, you get a straight line with the slope $4\pi^2/g$. You measure the period for five lengths and fit a line, as in chapter 3. Every measurement of $T$ has a random error of about 0.01 s. The question is: how sure is your value of $g$?

A simulation can answer it, because in a simulation we know the true value. We can make "measurements" with a known error, do the whole analysis, and see how far the answer is from the truth.

One experiment

Predict first. The true value is 9.81. Will the experiment give exactly this?

Repeat the whole experiment

In real life, you have one experiment. In a simulation, you can do it 2000 times.

Predict first. Will the average of all the values of $g$ be 9.81? How much will they scatter?

The histogram shows 1000 such experiments: a bell curve, centred on the true value, with a width of about 0.12. A good report of your one experiment would say: $g = 9.78 \pm 0.12$ m/s$^2$. The simulation told us what uncertainty to expect, before we even did the experiment. Scientists use this to plan experiments: should I use more lengths, or a better stopwatch?

The standard error of the mean

When you measure the same thing again and again, the mean of the measurements is more reliable than a single one. How much more? The standard error of the mean is

where $s$ is the sample standard deviation (with ddof=1) and $n$ is the number of measurements.

The formula agrees with the real spread, with some scatter of its own, because it comes from one small data set. The true value is $0.5 / \sqrt{n}$: 0.22, 0.11 and 0.056. To halve the uncertainty, you need four times as many measurements. This is the $1/\sqrt{N}$ law again, for the third time in this chapter.

Averaging only beats random scatter. A scale that always reads 2 g too much stays 2 g too much, however many weighings you average.

The bootstrap

What if there is no formula for the uncertainty of what you computed? The bootstrap is a clever way. You treat your data as the whole world: draw a new data set of the same size from your data, with replacement, compute the number, and repeat thousands of times. The spread of the computed numbers is the uncertainty.

A systematic error

Predict first. An instrument reads 0.3 too high. If you average a hundred thousand readings, will the mean be close to the truth?

Averaging removes random scatter but it cannot remove a systematic error. A very small standard error does not mean a correct answer: it only means a precise one. A systematic error makes the measurement inaccurate. The only cures are to calibrate the instrument against a known standard, or to measure the same thing with a different method.

Common mistakes

• Calling the standard deviation the uncertainty of the mean. The uncertainty of the mean is $s/\sqrt{n}$.
• Forgetting ddof=1. For a sample, divide by $n - 1$.
• Believing that more measurements remove every error. They remove only random error.
• Reporting a result without an uncertainty.
• Taking the uncertainty from one small data set as exact.

Practice

Start with two quick questions. Each New numbers gives you a fresh one, so you can practise as long as you like.

Then the labs and the problems:

1. Warm-up. Open the lab Standard error. You write the formula and the bootstrap.
2. Build. Open the lab Project: measure g and the problem How many measurements?
3. Stretch. Open the problem A biased instrument.

Remember these

• A random error is reduced by repeating. The standard error of the mean is $s/\sqrt{n}$, with ddof=1.
• To halve the uncertainty, use four times as many measurements.
• Simulate the whole experiment many times to find the spread of its answer. The bootstrap finds it from the data.
• A systematic error is not reduced by averaging. Calibrate the instrument.
• Report a result with its uncertainty: $9.78 \pm 0.12$.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. Ten measurements have $s = 0.4$. What is the standard error of the mean?

2. How many measurements make the standard error half as large?

3. What does the bootstrap do?

4. A scale always reads 2 grams too much. Will 10 000 weighings give the right answer?