A shuffling test for two groups

What you will be able to do

• explain why shuffling the group labels tests the null idea of no difference;
• find a p-value by shuffling, in a program;
• say what the result does and does not show.

Two groups of students

A teacher wants to know whether a new way of revising helps. She takes $16$ students and splits them at random into two groups of $8$. One group uses the new method, the other the old one. Then all $16$ take the same quiz, marked out of $20$. These are made-up scores.

• New method: $14$, $17$, $12$, $18$, $16$, $15$, $19$ and $13$ marks. Mean $15.5$ marks.
• Old method: $12$, $14$, $11$, $15$, $13$, $10$, $16$ and $12$ marks. Mean $12.875$ marks.

The new group's mean is $2.625$ marks higher. Is that the method, or just the luck of which students landed in which group?

The idea of shuffling

Take the null idea: the method makes no difference. Then each student would have scored the same in either group. The labels "new" and "old" would be like name tags handed out at random.

So we can imitate the null idea. Pool the $16$ scores, shuffle them, and deal them into two new groups of $8$. Work out the difference in means. Do it thousands of times. This builds the pattern of differences that luck alone makes. If the real difference of $2.625$ marks is rare in that pattern, luck is a poor explanation.

This test is called a shuffling test, or a randomisation test. It works because the groups were formed at random in the first place.

Try it in Python

The program shuffles the $16$ scores $5000$ times and counts the shuffles with a difference of $2.625$ marks or more.

Every split, counted

There are $\binom{16}{8} = 12\,870$ ways to split the $16$ scores into two groups of $8$. A computer can try every one. Exactly $309$ of them give a difference of $2.625$ marks or more. So the exact p-value is $\dfrac{309}{12\,870} \approx 0.0240$. The simulation's $0.023$ was close.

The picture shows all $12\,870$ differences. They pile up in a mound centred at zero, because a random split usually mixes the scores evenly. The real difference sits far out in the right tail.

The smooth curve is only a picture of the mound, so its p-value, about $0.02$, is close to the exact $0.0240$ but not equal to it.

What the result shows

The p-value is small, so luck alone is a poor explanation. Because the teacher assigned the students at random, the most likely cause of the difference is the method itself. You met random assignment in the chapter on study design.

Two cautions remain. The study used only $16$ students, so the size of the effect is uncertain. And the students all came from one class, so the result may not hold for other students.

Practise with new numbers

Common mistakes

• Shuffling when the groups were not formed at random. Then a small p-value cannot tell the method from other differences between the groups.
• Forgetting the direction. If you suspect the new method helps, count shuffles where the new group does at least as well.
• Treating the simulated p-value as exact. A simulation gives a close value that changes a little from run to run.

Remember these

• Under the null idea of no difference, the group labels mean nothing, so shuffling them imitates luck.
• The p-value is the share of shuffles at least as extreme as the real difference.
• With random assignment, a small p-value points to the treatment as the cause.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. Group A has scores $5$, $7$ and $9$. Group B has scores $3$, $4$ and $8$. Find the difference in means, A minus B.

2. Why is it fair to shuffle the labels when the null idea says the treatment makes no difference?

3. In $4000$ shuffles, $280$ give a difference at least as large as the real one. Estimate the p-value. Is luck a good explanation?

4. Two groups of $5$ students each are to be compared. How many ways are there to split the $10$ scores into two groups of $5$?

5. A study compares people who chose to drink green tea with people who did not. A shuffling test gives a small p-value. Does green tea cause the difference?

References

• OpenIntro, OpenIntro Statistics, Chapter 2 case study and Chapter 5, on randomisation tests.
• OpenStax, Introductory Statistics, Chapter 10, on comparing two groups.