Simpson's paradox

What you will be able to do

• work out the pass rates inside groups and overall from a two-group table;
• explain how combining groups can reverse a comparison;
• decide whether the group-by-group or the combined comparison answers the question.

Two revision methods

A made-up tutoring centre tries two revision methods for a test, A and B, with $200$ students each. Students who started with low marks find the test harder than students who started with high marks. Here are the results.

Find the pass rates inside each group.

• Low starters: A has $\dfrac{60}{150} = 40\%$ and B has $\dfrac{18}{50} = 36\%$. A is better.
• High starters: A has $\dfrac{45}{50} = 90\%$ and B has $\dfrac{126}{150} = 84\%$. A is better again.

Now put the groups together. A has $60 + 45 = 105$ passes out of $200$, which is $52.5\%$. B has $18 + 126 = 144$ passes out of $200$, which is $72\%$. Overall, B is better.

A wins in both groups and loses overall. This is Simpson's paradox, named after the statistician Edward Simpson, who wrote about it in 1951.

Why it happens

Look at who took each method. Three quarters of A's students were low starters, who pass less often whatever the method. Three quarters of B's students were high starters, who pass more often. The overall rate of A is pulled down by its many low starters. It is an average of $40\%$ and $90\%$, weighted heavily towards $40\%$. B's overall rate is weighted towards $84\%$.

The starting level is a lurking variable. It affects both the method a student ended up with and the chance of passing.

Which answer is right?

Ask what you want to know. Here the question is: for a student like me, which method is better? A student is either a low starter or a high starter, and for both, A is better. So the group-by-group comparison answers the question. The overall comparison mixes two different kinds of student in different amounts.

The fair cure is a design that puts the same mix of students into each method, such as random assignment. Then the overall comparison and the group comparison agree.

Not every combined table is misleading. If the groups are mixed in the same proportions in both methods, the paradox cannot happen.

Your turn: new numbers

Try it in Python

The program finds the pass rate of each method inside each group, and then overall.

The usual mistakes

• Averaging percentages from groups of different sizes. Weight each group by its size, or count passes and totals.
• Trusting the combined table automatically. Check whether the groups are mixed in different proportions.
• Thinking the paradox is rare. It appears whenever a lurking variable is linked to both the grouping and the outcome.
• Thinking the group comparison is always right. It is right when the groups match the question you are asking.

Remember these

• Simpson's paradox: a comparison holds in every group but reverses when the groups are combined.
• It happens when the groups are mixed in different proportions and the groups have very different rates.
• Compare within groups when the grouping variable affects the outcome.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. In one group, $30$ of $40$ people say yes. In another, $10$ of $60$ say yes. What percentage of all $100$ people say yes?

2. Why did method A lose overall, though it won in both groups?

3. Could Simpson's paradox happen if both methods had $100$ low starters and $100$ high starters?

4. A parent says the overall table shows that method B is better for her child, who is a high starter. Is she right?

5. What is the lurking variable in the tutoring centre?

References

• OpenStax, Introductory Statistics, Chapter 3, the section on contingency tables.