Screening tests and other traps

What you will be able to do

• use sensitivity, specificity and the positive and negative predictive values;
• spot the base rate fallacy and the prosecutor's fallacy;
• solve the two-child puzzle and the game with three doors, and update a probability with a second piece of evidence.

Four numbers for a test

A medical test is described by four probabilities.

• Sensitivity: $P(\text{positive} \mid \text{ill})$. The test finds the ill people.
• Specificity: $P(\text{negative} \mid \text{healthy})$. The test clears the healthy people.
• Positive predictive value: $P(\text{ill} \mid \text{positive})$. What a positive result means for you.
• Negative predictive value: $P(\text{healthy} \mid \text{negative})$. What a negative result means for you.

The first two belong to the test. The last two also depend on how common the illness is. A-Level, IB and AP talk about false positives and false negatives: a false positive is a positive result for a healthy person, and its chance is $1 - \text{specificity}$.

The base rate fallacy

Take the town of the last lesson: $1\%$ ill, sensitivity $0.95$, specificity $0.90$. We found $P(\text{ill} \mid +) \approx 0.088$. The negative result is much more reassuring:

The base rate fallacy is to ignore how common the illness is and to read "the test is $95\%$ accurate" as "the person is $95\%$ likely to be ill". The curve in the simulation of the last lesson shows how strongly the answer depends on the base rate.

The prosecutor's fallacy

A profile found at a crime scene matches one person in $10\,000$ by chance. A suspect is chosen from a town of $100\,000$ people with no other evidence, and the suspect matches. The chance that a match happens to an innocent person is $\dfrac{1}{10\,000}$. A careless argument says the suspect is therefore guilty with probability $0.9999$. This confuses $P(\text{match} \mid \text{innocent})$ with $P(\text{innocent} \mid \text{match})$.

The correct reasoning uses Bayes' theorem. Among $100\,000$ people, one is the true source, and about $10$ more match by chance. So of the roughly $11$ people who match, only one is the source. Exactly: $P(\text{source} \mid \text{match}) = \dfrac{1}{1 + 99\,999/10\,000} = \dfrac{10\,000}{109\,999} \approx 0.091$. This is an illustration of the logic. A real case has other evidence, and a court weighs all of it.

Two children

A family has two children. Each child is equally likely to be a girl or a boy, independently.

• At least one is a girl. The equally likely cases are GG, GB and BG, so $P(\text{both girls}) = \dfrac{1}{3}$.
• The elder child is a girl. The cases are GG and GB, so $P(\text{both girls}) = \dfrac{1}{2}$.

The two pieces of news look alike, but they remove different outcomes from the sample space. Always list the sample space and cross out only what the news rules out.

The game with three doors

A car is behind one of three doors, and a goat is behind each of the others. You pick door $1$. The host knows where the car is. He opens another door with a goat, and he offers you a switch. Should you switch?

Use Bayes' theorem. Suppose the host opens door $3$. The car is behind door $1$, $2$ or $3$ with probability $\dfrac{1}{3}$ each.

1. Car behind door 1. The host may open door $2$ or door $3$, so he opens door $3$ with probability $\dfrac{1}{2}$. The path has probability $\dfrac{1}{3} \times \dfrac{1}{2} = \dfrac{1}{6}$.
2. Car behind door 2. The host must open door $3$. The path has probability $\dfrac{1}{3} \times 1 = \dfrac{1}{3}$.
3. Car behind door 3. The host cannot open it. The path has probability $0$.
4. The posterior. $P(\text{car at door } 2 \mid \text{host opens } 3) = \dfrac{1/3}{1/6 + 1/3} = \dfrac{2}{3}$.

So switching wins with probability $\dfrac{2}{3}$ and staying wins with probability $\dfrac{1}{3}$. The trap is to think that two closed doors mean a fifty-fifty chance. The host's choice was not random, so the doors are not equally likely.

A second test

Use the posterior of one test as the prior of the next. This is right when the tests are independent given the illness. Take a disease with prevalence $10\%$, and a test with sensitivity $90\%$ and specificity $90\%$.

1. After one positive. $\dfrac{0.1 \times 0.9}{0.1 \times 0.9 + 0.9 \times 0.1} = \dfrac{1}{2}$.
2. After a second positive. The new prior is $\dfrac{1}{2}$, so $\dfrac{0.5 \times 0.9}{0.5 \times 0.9 + 0.5 \times 0.1} = \dfrac{9}{10}$.

One positive test raised the chance from $10\%$ to $50\%$, and a second one raised it to $90\%$.

Practise with new numbers

Try it in Python

The program plays the game with three doors. It first weighs every case exactly. Then it plays $30\,000$ games with a fixed seed, so the numbers repeat each time you run it.

Common mistakes

• Reading the sensitivity as the answer. "The test is $95\%$ accurate" does not mean "a positive result is $95\%$ likely to be right".
• Ignoring the base rate. The rarer the illness, the more a positive result is likely to be a false alarm.
• Treating "at least one" as "a particular one". The two-child answers differ because the news is different.
• Saying two doors are fifty-fifty. The host's action is not random, so the two closed doors are not equally likely.

Remember these

• Sensitivity is $P(+ \mid \text{ill})$. Specificity is $P(- \mid \text{healthy})$. The predictive values turn them around.
• A positive result for a rare illness is often a false alarm.
• $P(\text{evidence} \mid \text{innocent})$ is not $P(\text{innocent} \mid \text{evidence})$.
• List the sample space, cross out what the news rules out, and count what is left.
• With independent tests, the posterior of one test is the prior of the next.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. An illness affects $1$ person in $1000$. A test has sensitivity $0.99$ and specificity $0.99$. What is the chance that a person with a positive test is ill?

2. A family has two children, and each child is equally likely to be a boy or a girl. You learn that at least one child is a boy. What is the probability that both are boys?

3. In a game with $4$ doors, one has a car. You pick a door. The host, who knows where the car is, opens two other doors with goats. You switch to the last closed door. What is the probability that you win?

4. A lawyer says: "The chance of this match happening by accident is $\dfrac{1}{10\,000}$, so the suspect is innocent with probability $\dfrac{1}{10\,000}$." What is wrong with this?

5. A disease has prevalence $10\%$. A test has sensitivity $90\%$ and specificity $90\%$. A person tests positive twice, with the two tests independent. What is the probability that the person is ill?

References

• OpenStax, Introductory Statistics, Chapter 3, section on contingency tables, and the section on probability topics with its worked problems.
• NCERT, Mathematics Part II for Class XII, Chapter 13, section on Bayes' theorem.