Significance, errors and honest reporting

What you will be able to do

• use a significance level to make a decision, and word the decision correctly;
• name the two kinds of error;
• explain why many tests produce false alarms, and how honest reports deal with it.

Drawing a line

Before looking at the data, a researcher chooses a significance level, written $\alpha$ (alpha). The usual choice is $\alpha = 0.05$.

• If the p-value is at most $\alpha$, the result is statistically significant. We reject the null idea.
• If the p-value is larger than $\alpha$, we do not reject the null idea.

Notice the words. We never "accept" or "prove" the null idea. Not rejecting it only means the evidence was not strong enough. "Significant" here means "hard to explain by luck", not "important".

Two kinds of error

A decision can be wrong in two ways.

1. A Type I error is a false alarm: we reject a null idea that is true. If the null idea is true, this happens with probability $\alpha$. With $\alpha = 0.05$, one test in twenty of a true null idea raises a false alarm.
2. A Type II error is a missed effect: we do not reject a null idea that is false. Small samples make this likely. The chance of catching a real effect is called the power of the test.

A smaller $\alpha$ means fewer false alarms but more missed effects. For a fixed sample, you cannot shrink both at once. A larger sample helps with both.

What a p-value is not

A p-value is easy to misread. It is not any of these.

1. The chance that the null idea is true. It is a chance about the data, worked out by assuming the null idea.
2. The size of the effect. A tiny difference can give a tiny p-value when the sample is huge.
3. A measure of importance. A significant result can be too small to matter in practice.

Many tests find "something"

Suppose a study tests $20$ ideas, and in fact every null idea is true. Each test has a chance $0.05$ of a false alarm. The chance that all $20$ stay quiet is $0.95^{20} \approx 0.358$. So the chance of at least one "significant" result is about $0.64$.

Run enough tests and something will look significant, by luck alone. A report that shows only the one significant test, and hides the other nineteen, misleads its readers.

Try it in Python

The program imitates many groups of twenty studies. Each study tosses a fair coin $100$ times and calls the coin unfair if it sees $60$ or more heads, or $40$ or fewer. Every coin is fair, so every "find" is a false alarm.

Honest reporting

Good researchers protect their readers from false alarms. They

1. decide the question, the test and $\alpha$ before they see the data;
2. report every test they ran, not only the significant ones;
3. give the size of the effect with a confidence interval, not only a p-value;
4. repeat a surprising study before they trust it.

One simple guard for many tests divides $\alpha$ by the number of tests. With $10$ tests, each test uses $0.005$. This is called the Bonferroni correction.

Practise with new numbers

Common mistakes

• Saying "we accept the null idea". Say "we do not reject it".
• Reading "significant" as "important". It only means hard to explain by luck.
• Reading the p-value as the chance that the null idea is true.
• Reporting only the tests that worked. Many tests make false alarms almost certain.

Remember these

• Reject $H_0$ when the p-value is at most $\alpha$, usually $0.05$.
• Type I error: a false alarm, with probability $\alpha$ when $H_0$ is true. Type II error: a missed effect.
• Among $m$ independent tests of true null ideas, the chance of at least one false alarm is $1 - (1 - \alpha)^m$.
• Report every test, the size of the effect and an interval.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. A test at $\alpha = 0.05$ gives a p-value of $0.03$. What is the decision?

2. A new medicine does nothing, but a trial concludes that it works. Which kind of error is this?

3. A small study fails to detect a real benefit of a new diet. Which kind of error is this, and what would make it less likely?

4. A survey tests $10$ independent ideas at $\alpha = 0.05$, and all ten null ideas are true. Find the chance of at least one false alarm.

5. A report says: "p = 0.001, so there is only a $0.1\%$ chance that the new method does nothing." What is wrong?

References

• OpenStax, Introductory Statistics, Chapter 9, sections on outcomes and the Type I and Type II errors.