Non-response, wording and sample size

What you will be able to do

• work out the share of the replies when two groups reply at different rates;
• spot a leading, loaded or double question and rewrite it fairly;
• explain why the quality of a sample matters more than its size.

The form that came back

A council sends a form about a new bus route to 1000 adults chosen at random. Only 300 send it back. The sample was chosen fairly. Is the result fair?

Not always. It depends on who sent the form back. If the busy people threw it away, and busy people need the bus most, the replies have a slant.

Non-response

Non-response happens when people chosen for a sample do not answer. The response rate is the share who do answer: here $300/1000 = 30\%$. A low response rate is a warning sign, but the real danger is that the people who answer differ from the people who do not.

Here is a worked example, all made up. In a town, 50% of the adults support a plan. Supporters reply to a survey 20% of the time, and the others 40% of the time.

1. Think of 100 adults. 50 supporters, so $50 \times 0.2 = 10$ replies. 50 others, so $50 \times 0.4 = 20$ replies.
2. The replies. 30 forms, and 10 of them support the plan.
3. The result. $\dfrac{10}{30} \approx 33.3\%$, against a truth of 50%.

The sample was random, and still the result is about 17 percentage points too low.

Words that push

The wording of a question can change the answer. This is response bias.

• Leading question. "Do you agree that the dangerous crossing needs a traffic light?" The word "dangerous" pushes towards yes. Better: "Should the crossing on Hill Road have a traffic light?"
• Loaded words. "Should the council waste money on a new stadium?" Better: "Should the council spend money on a new stadium?"
• Two questions in one. "Is the canteen food tasty and cheap?" A student who finds it tasty but costly cannot answer. Ask two questions.
• Answers people are shy to give. Asked face to face, people say they read more books and eat less sugar than they really do.

Course 3 of this chapter shows how to write a good survey question from scratch.

Bigger is not better: a famous poll

In 1936 a magazine in the United States, the Literary Digest, mailed about 10 million voting forms to predict the election for President. About 2.4 million came back. The magazine predicted a clear win for Alf Landon. In fact Franklin Roosevelt won by a landslide, with about 61% of the vote.

What went wrong? The names came largely from lists such as telephone books, car owners and the magazine's own readers. Many poorer voters were missing. Also, fewer than one in four people replied, and the people who replied leaned one way. A poll with millions of replies was still biased, in both who was asked and who answered. Smaller polls with better samples called the result correctly.

The band of luck shrinks as the replies grow, but it shrinks around the red line, not the green one. A huge sample makes you very sure of the wrong answer.

Practise with new numbers

Try it in Python

The program runs the made-up poll of the worked example: supporters reply 2 times in 10, and the others 4 times in 10. It asks more and more people at random.

Your turn.

The usual mistakes

• Trusting a large number of replies. Ask how the people were chosen and who replied. The count alone tells you nothing about bias.
• Ignoring the people who did not answer. The question is not "how many replied?" but "are the people who replied like the people who did not?".
• Writing a question that hints at the answer. Use neutral words, and ask one thing at a time.
• Thinking a random sample is safe whatever happens next. Non-response and wording can spoil a sample that was chosen fairly.

Remember these

• The response rate is the share who answer. A low rate is a warning.
• Share of supporters in the replies $= \dfrac{p\,r_y}{p\,r_y + (100 - p)\,r_n} \times 100\%$.
• Leading, loaded and double questions push the answers. Neutral wording helps.
• Size reduces luck, never bias. Quality first, then size.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. A club emails 250 members, and 85 reply. Find the response rate.

2. In a town, 40% of the adults support a plan. Supporters reply 50% of the time and the others 25% of the time. What percentage of the replies support the plan?

3. Rewrite this question fairly: "Do you agree that the noisy new factory should close at night?"

4. What is wrong with the question "Is the school library quiet and well stocked?"

5. Poll A has $20\,000$ replies to a phone-in. Poll B has 800 replies from a random sample with a high response rate. Which do you trust more, and why?

References

• OpenStax, Introductory Statistics, Chapter 1, section on data, sampling and variation, which discusses sampling errors and bias.
• Many statistics texts tell the story of the 1936 Literary Digest poll. The figures here are the ones usually quoted.