Designing a fair experiment

What you will be able to do

• design an experiment with a control group and random assignment;
• explain what a placebo and blinding guard against;
• work out the effect of a treatment from the results of the two groups.

A new plant food

A gardening shop sells a new plant food. It says, "Plants grow 20% taller!" You give it to your tomato plants, and they grow tall. Does that prove anything?

Not yet. Perhaps it was a sunny summer. Perhaps you watered more because you were watching. To know what the plant food does, you need plants that did not get it, grown in the same way at the same time.

The parts of a fair experiment

1. A control group. A group that gets no treatment, or the usual one. It shows what happens without the new treatment. The new treatment is then judged by the difference between the groups.
2. Random assignment. A chance device, such as a coin or a shuffle, decides which subjects get the treatment. Then the groups are alike on average in every way, even in ways nobody measured. This is the big idea of the course.
3. A placebo. In trials on people, the control group gets a dummy treatment that looks the same, such as a sugar pill. People often feel better just because they were treated. This is the placebo effect, and the placebo makes it the same in both groups.
4. Blinding. In a single-blind experiment the subjects do not know their group. In a double-blind experiment the people who give the treatment and measure the results do not know either. Then hopes cannot colour the results.
5. Enough subjects. With only 3 plants in each group, luck can swamp the effect. Repeating the treatment on many subjects is called replication.

A note on care. Experiments on people need their informed consent: they agree after being told the purpose and the risks. No one may be given a treatment that is known to harm.

A worked example

A made-up trial tests a cream for a skin rash. By random assignment, 200 patients get the cream and 200 get a cream with no medicine in it. Neither the patients nor the nurses who check the rash know who got which.

1. Medicine group. 124 of 200 improve, which is 62%.
2. Placebo group. 90 of 200 improve, which is 45%.
3. The effect. $62 - 45 = 17$ percentage points.

Without the placebo group, the company might claim that the cream helps 62% of patients. Most of those patients would have improved with any cream.

Groups that are alike: blocks

Sometimes we know a variable matters, such as the age of a patient or the field a plant grows in. Then we can put alike subjects together in blocks and assign the treatment at random inside each block. This is the experiment's version of a stratified sample. A matched pairs design is the smallest block: two alike subjects, and a coin decides which gets the treatment.

Practise with new numbers

Try it in Python

The program uses 40 made-up volunteers for a trial, with their ages. It compares a poor plan, the first 20 to sign up get the treatment, with random assignment done 1000 times.

Your turn.

The usual mistakes

• No control group. A treatment that "works" may only show what happens anyway.
• Letting people choose their group. Then the groups differ in more than the treatment. Assign at random.
• Forgetting the placebo effect. Feeling treated can help on its own.
• Measuring with hope. If the person who measures knows the groups, use double blinding.

Remember these

• Control group, random assignment, placebo, blinding, enough subjects.
• The effect is the difference between the treatment group and the control group.
• Random assignment makes the groups alike on average, even in things nobody measured.
• Blocks and matched pairs are the experiment's version of strata.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. Why does a trial of a new pain tablet give the control group a sugar pill, and not nothing at all?

2. In a trial, neither the patients nor the doctors who check them know who got the real medicine. What is this called?

3. In a trial, 65% of the treatment group and 40% of the placebo group improve. What is the effect of the treatment?

4. A teacher lets students choose whether to try a new spelling game, then compares marks. What is the main flaw?

5. A scientist tests a plant food on 30 plants in a field where one end is wetter than the other. How can she use blocks?

References

• OpenStax, Introductory Statistics, Chapter 1, section on experimental design and ethics, which covers control groups, placebos and blinding.