Surveys, observational studies and experiments

What you will be able to do

• tell a sample survey, an observational study and an experiment apart;
• name the explanatory variable and the response variable;
• write a survey question that is clear and fair, with answer choices that work.

Green tea and a long life

A news story says, "People who drink green tea live longer." How did the researchers find that out? Perhaps they asked people about their habits and followed them for years. Nobody told anyone to drink tea. The tea drinkers chose it themselves. Keep that in mind. It is the key to this whole course.

Three kinds of study

• Sample survey. Ask a sample of people questions, such as "How many hours do you sleep?". The first two courses of this chapter were about choosing that sample.
• Observational study. Record what people already do, and what happens to them. Nobody is told what to do.
• Experiment. The researcher imposes a treatment on some subjects, and measures the result. The people, animals or things in an experiment are its subjects (or experimental units).

A study that watches students who already use a revision app is observational. A study that gives the app to some students, chosen by the researcher, is an experiment.

Explanatory and response variables

The explanatory variable is the one we think may explain or change the other. In an experiment it is the treatment. The response variable is the outcome we measure.

• Does the amount of sleep change test marks? Explanatory: hours of sleep. Response: test mark.
• Does a fertiliser make tomato plants grow taller? Explanatory: fertiliser or not. Response: height of a plant, in cm.

Some books call them the independent and the dependent variable. They mean the same thing.

Why only an experiment shows a cause

In an observational study, the people choose for themselves. Students who choose a revision app may be keener than the others, and keen students score more marks anyway. So the gap between the two groups mixes the effect of the app with the effect of being keen. In an experiment the researcher decides who gets the app, so being keen is no longer tied to the app.

Observational studies are still useful. Some questions cannot be tested by an experiment: we cannot tell people to smoke for twenty years. Then careful observational studies are the best we have, and the next lessons show how to read them.

Writing a good survey question

A survey is only as good as its questions. Here is a checklist.

1. One idea at a time. Not "Is the bus quick and clean?".
2. Plain, neutral words. No "Do you agree that...", and no words that hint at an answer.
3. A clear time and unit. Not "Do you sleep a lot?" but "How many hours did you sleep last night?".
4. Answer choices that cover every case and do not overlap. Ages "10 to 20, 20 to 30" put a 20-year-old in two boxes. Use "10 to 19, 20 to 29, 30 or over".
5. Open or closed, on purpose. A closed question gives choices. It is easy to count. An open question lets people answer in their own words. It is richer but harder to count.
6. Try it first. Ask a few people in a small trial, called a pilot, and fix what confuses them.

Practise with new numbers

Try it in Python

The program builds $2000$ made-up students, half of them keen. The app really adds 3 marks. In the observational study, students choose the app for themselves. In the experiment, a coin decides.

Your turn.

The usual mistakes

• Calling every study an experiment. If nobody was given a treatment by the researcher, it is observational.
• Swapping the two variables. The explanatory variable comes first in time and in the question. The response is what we measure afterwards.
• Reading a cause into an observational study. It shows a link, not a cause.
• Overlapping answer choices. Check the ends of every range.

Remember these

• Survey: ask. Observational study: watch. Experiment: impose a treatment.
• Explanatory variable: the possible cause. Response variable: the outcome.
• Only a well-designed experiment can show a cause.
• A good question asks one thing, in neutral words, with choices that cover every case once.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. A researcher records how many hours of television 300 children watch and their reading scores. Is this an observational study or an experiment?

2. A farmer sprays half of her fields with a new spray and measures the yield of every field. Name the explanatory and the response variable.

3. What is wrong with the answer choices "less than 1 hour, 1 to 2 hours, 2 to 3 hours, more than 3 hours"?

4. Rewrite this question fairly: "How much do you love our new menu?"

5. Students who sit in the front row get higher marks. Why can we not say that the front row causes higher marks?

References

• OpenStax, Introductory Statistics, Chapter 1, section on experimental design and ethics.