Testing an Idea with Simulation: Quick Revision

A brief note on Testing an Idea with Simulation. Read it before a test. Try each question at the end before you open its answer.

Words and rules

• Null idea $H_0$: nothing special is going on (a fair coin, no difference).
• Alternative $H_a$: what we suspect. One-sided ($p > 0.5$) or two-sided ($p \ne 0.5$), chosen before seeing the data.
• p-value: if $H_0$ were true, the chance of a result at least as extreme as the one seen.
• By simulation: the share of simulated results at least as extreme.
• Shuffling test: pool two groups, shuffle the labels, and count the shuffles at least as extreme.
• Decision: reject $H_0$ when the p-value is at most $\alpha$, usually $0.05$. Otherwise do not reject it.
• Type I error: a false alarm, with probability $\alpha$ when $H_0$ is true.
• Type II error: a missed effect. A larger sample makes it rarer.
• Many tests: $1 - (1 - \alpha)^m$ is the chance of at least one false alarm.

A worked example

A fair coin is tossed $20$ times, and $15$ heads appear.

1. Ideas. $H_0$: $p = 0.5$. $H_a$: $p > 0.5$.
2. p-value. Count the results with $15$ or more heads: $15\,504 + 4845 + 1140 + 190 + 20 + 1 = 21\,700$. Then $P(X \ge 15) = \dfrac{21\,700}{2^{20}} \approx 0.0207$.
3. Decision. $0.0207 \le 0.05$: reject $H_0$ at the $0.05$ level.

Practice with new numbers

A program to try

The program finds the p-value of the worked example and the chance of a false alarm among $20$ tests.

Mistakes to avoid

• Reading the p-value as the chance that $H_0$ is true.
• Saying "we accept $H_0$". Say "we do not reject it".
• Calling a significant result important. Check the size of the effect.
• Hiding the tests that did not work.

Check yourself

1. In $1000$ shuffles, $8$ are at least as extreme as the real difference. What is the p-value?

2. A test at $\alpha = 0.05$ gives a p-value of $0.07$. What is the decision?

3. A true null idea is rejected. Which error is this?

4. Why must the groups of a shuffling test be formed at random to speak of cause?

5. Find the chance of at least one false alarm among $5$ independent tests at $\alpha = 0.05$.

Sources

• OpenStax, Introductory Statistics, Chapter 9, hypothesis testing with one sample.
• OpenIntro, OpenIntro Statistics, Chapter 5, foundations for inference.