Is the coin fair? A test by simulation

What you will be able to do

• write the null idea and the alternative idea for a question;
• find a p-value by simulation and by exact counting;
• say in plain words what a p-value means.

A suspicious coin

A friend tosses her coin $50$ times and gets $34$ heads. A fair coin should give about $25$. There are two ways to explain the result.

1. The coin is fair, and she was lucky.
2. The coin really does favour heads.

We cannot toss the coin forever. But we can ask a sharp question: if the coin were fair, how often would $50$ tosses give $34$ heads or more? If that almost never happens, luck is a poor explanation.

Two ideas

A test always starts from two ideas about the population.

• The null idea, written $H_0$, says that nothing special is going on. Here: the coin is fair, $p = 0.5$.
• The alternative idea, written $H_a$ or $H_1$, is what we suspect. Here: the coin favours heads, $p > 0.5$.

Books call these the null hypothesis and the alternative hypothesis. We assume the null idea and see whether the data look strange under it. The test never assumes what we hope to show.

Try it in Python

The program tosses a fair coin $50$ times, ten thousand times over, and counts how often it gets $34$ heads or more. Then it works out the same chance exactly.

The p-value

The share we just found is the p-value. In plain words:

If the null idea were true, the p-value is the chance of getting a result at least as extreme as the one we saw.

For the coin, the exact p-value is $P(X \ge 34) \approx 0.0077$. The picture shows where it comes from. The orange bars are the results of $34$ heads or more. Together they hold less than one percent of the chance.

A small p-value means the data would be rare if the null idea were true. So either something rare happened, or the null idea is wrong. With a p-value under one in a hundred, most people would doubt that the coin is fair.

Now suppose the friend had got $29$ heads. Then $P(X \ge 29) \approx 0.161$. A fair coin does that about one time in six. That is no reason to doubt the coin.

One side or two

Our friend suspected heads, so we counted only results with many heads. This is a one-sided test. If she had asked "is the coin unfair in either direction?", we would also count results with $16$ heads or fewer. For a fair coin the two tails are equal, so the two-sided p-value is twice as large, about $0.0153$. Decide which question you are asking before you look at the data.

Practise with new numbers

Common mistakes

• Counting only the result seen. The p-value counts the result seen and every result more extreme.
• Putting what we hope to show in the null idea. The null idea is the "nothing special" idea.
• Choosing one side after seeing the data. That halves the p-value unfairly.
• Saying the null idea is proved when the p-value is large. A large p-value only says that the data fit the null idea.

Remember these

• The null idea $H_0$ says nothing special is going on. The alternative $H_a$ is what we suspect.
• The p-value is the chance, if $H_0$ were true, of a result at least as extreme as the one seen.
• A simulation finds it as a share of simulated results. Counting finds it exactly.
• A small p-value is evidence against $H_0$.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. A cafe claims that $70\%$ of its customers order tea. A student suspects that fewer do. Write the null idea and the alternative idea.

2. In $2000$ simulated sets under the null idea, $46$ are at least as extreme as the real result. Estimate the p-value.

3. A fair coin is tossed $5$ times. What is the probability of $5$ heads? What p-value does a person who saw $5$ heads in $5$ tosses get?

4. Why do we count results more extreme than the one seen, and not only the result itself?

5. A test gives a p-value of $0.45$. Does this prove that the null idea is true?

References

• OpenStax, Introductory Statistics, Chapter 9, sections on null and alternative hypotheses and on the p-value.
• OpenIntro, OpenIntro Statistics, Chapter 5, on hypothesis testing and simulation.