Selection bias: who gets left out

What you will be able to do

• explain the difference between bias and bad luck;
• spot a convenience sample, a voluntary response sample, undercoverage and survivorship bias;
• work out the size of a bias when you know how each group feels.

Text in your answer

A radio show asks, "Should our town ban cars from the market street? Text YES or NO." By the evening, 9 in 10 texts say YES. Does the town agree?

We cannot tell. Who listens to this show? Who bothers to text? People with strong views text, and people who do not care stay quiet. The sample chose itself, so it speaks only for the people who chose to answer.

Bias is not bad luck

In the last lesson a random sample missed the truth by luck. Sometimes it was too high and sometimes too low. On average it was right.

A biased method misses in the same direction every time. If you repeated it with a thousand new samples, the results would still crowd around the wrong value. Bias comes from how the sample was chosen, so a larger sample does not cure it.

The usual kinds of selection bias

• Convenience sample. You ask the people who are easiest to reach: friends, the people at one shop, the first 50 on a list. They may not be like everyone else.
• Voluntary response sample. People choose themselves, as on the radio show or an online poll. Strong opinions answer more often, and usually the angry ones most of all.
• Undercoverage. Some groups can never be chosen. A phone survey misses people with no phone. An online survey misses people who are rarely online. A list of house owners misses everyone who rents.
• Survivorship bias. You look only at the things that came through some filter. The ones that failed have gone, so you never see them.

Survivorship bias, more closely

A town has a few houses that are 150 years old, and all of them are solid. "Old houses were built better," someone says. But the weak old houses fell down long ago. We see only the survivors.

A well-known story from the Second World War makes the same point. Engineers studied the bullet holes on bombers that came back from missions, and planned to add armour where the holes were thickest. A statistician pointed out the opposite. The planes hit in the other places had not come back at all, so those places needed the armour. The story is told in many versions, but the lesson is the same.

How large is a bias?

Here is a made-up town. Of the adults, 30% use the library. Of the library users, 80% want the library to stay open later. Of the other adults, 40% want it. What is the truth for the whole town?

1. Weight each group by its size. $\dfrac{30 \times 80 + 70 \times 40}{100} = \dfrac{2400 + 2800}{100} = 52\%$.
2. What the library survey finds. It asks library users only, so it finds about 80%.
3. The bias. $80 - 52 = 28$ percentage points.

A percentage point is the gap between two percentages. From 52% to 80% is 28 percentage points.

The library survey is right only when the two groups feel the same, or when nearly everyone uses the library. Try $a = b$ in the simulation.

Practise with new numbers

Try it in Python

The program builds the made-up town of $10\,000$ adults. Then it asks 200 people at the library and 200 people chosen at random from the whole town.

Your turn.

The usual mistakes

• Thinking a big sample cannot be biased. Size cures bad luck, not bias. The next lesson shows a famous poll with millions of replies that was badly wrong.
• Calling every unusual result "bias". A random sample that happens to come out high is unlucky, not biased. Bias is about the method.
• Forgetting who was never asked. Ask "who could not be in this sample?" as well as "who is in it?".
• Mixing up percent and percentage points. A change from 40% to 50% is 10 percentage points, but it is a rise of 25 percent.

Remember these

• Bias is a slant in the method. It pushes every sample the same way.
• Convenience, voluntary response, undercoverage and survivorship are the usual kinds of selection bias.
• Truth for a town $= \dfrac{f a + (100 - f) b}{100}$, where $f\%$ are in one group.
• A larger biased sample is only more sure of the wrong answer.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. A website asks its readers, "Is homework a waste of time?", and 3000 people click an answer. Which kind of bias is most likely?

2. To learn how far students travel to school, Mei asks the students in the school bus queue. What kind of sample is this, and which way is it likely to be wrong?

3. A survey of a town uses a list of the people who own a car. Which group does it leave out?

4. A magazine studies 25 famous companies and finds that all of them had a bold boss. It says that a bold boss makes a company succeed. What is missing?

5. In a town, 50% of the adults are gardeners. Of the gardeners, 90% want a new park. Of the others, 30% want it. A survey asks gardeners only. Find the bias in percentage points.

References

• OpenStax, Introductory Statistics, Chapter 1, sections on data, sampling and variation in data and sampling.