Bayes' theorem

What you will be able to do

• reverse a conditional probability with Bayes' theorem;
• name the prior, the likelihood and the posterior;
• check a Bayes answer with natural frequencies.

Which machine made the bolt?

In the last lesson we found that $2.9\%$ of all bolts are faulty. Now a faulty bolt is lying on the table. Which machine made it?

A first guess is $M_3$, because it has the highest fault rate. But $M_3$ makes only a fifth of the bolts. A second guess is $M_1$, because it makes the most bolts. Both guesses use only half of the information. Bayes' theorem uses both halves.

The theorem

Let $E_1, \dots, E_n$ be a partition and let $A$ be an event with $P(A) > 0$. Start from the definition of conditional probability and use the multiplication rule on the top. Then use the law of total probability on the bottom:

Three names are worth knowing.

• The prior $P(E_i)$ is the chance of the cause before you see the result.
• The likelihood $P(A \mid E_i)$ is the chance of the result if that cause is true.
• The posterior $P(E_i \mid A)$ is the chance of the cause after you see the result.

Indian books call the prior and posterior probabilities "a priori" and "a posteriori". A-Level, IB and AP use prior and posterior. The bottom of the fraction is the total probability of $A$, which is the sum of all the paths that end in $A$. The top is one of those paths.

The machines again

The three paths ending in "faulty" are $0.01$, $0.009$ and $0.01$, and they add up to $0.029$. The posterior of each machine is its path divided by $0.029$:

The three posteriors add up to $1$, as they must, because the faulty bolt came from exactly one machine. Notice that $M_1$ and $M_3$ are equally likely. $M_1$ makes more bolts, and $M_3$ makes worse bolts, and the two effects cancel.

Natural frequencies

Many people find Bayes' theorem easier with counts. Imagine $10\,000$ bolts. Then $M_1$ made $5000$ of them and $100$ of those are faulty. $M_2$ made $3000$ and $90$ are faulty. $M_3$ made $2000$ and $100$ are faulty. In all, $290$ bolts are faulty. Of these $290$, exactly $100$ came from $M_1$, so the chance is $\dfrac{100}{290} = \dfrac{10}{29}$. Same answer, no formula.

A screening test

In a town, $1$ person in $100$ has a certain illness. A test finds $95\%$ of the people who have the illness. It wrongly flags $10\%$ of the healthy people. Take $10\,000$ people.

1. Ill people. $100$ are ill. The test flags $95$ of them.
2. Healthy people. $9900$ are healthy. The test flags $990$ of them.
3. All positive tests. $95 + 990 = 1085$.
4. The posterior. Of the $1085$ people with a positive test, only $95$ are ill. So $P(\text{ill} \mid +) = \dfrac{95}{1085} = \dfrac{19}{217} \approx 0.0876$.

A positive result from a good test leaves less than a $9\%$ chance of being ill. The reason is the size of the groups. There are so many more healthy people that even a small error rate produces more false alarms than true ones.

Practise with new numbers

Try it in Python

The program has one function that does Bayes' theorem for any partition. It is used twice: once for the three machines and once for the screening test.

Common mistakes

• Turning the condition around. $P(\text{positive} \mid \text{ill})$ is not $P(\text{ill} \mid \text{positive})$. They are linked by Bayes' theorem.
• Forgetting the prior. A rare cause stays unlikely even when a test points to it.
• A wrong denominator. The bottom is the sum of all the paths that end in $A$, not just the one path on top.
• Mixing percentages and fractions. Change everything to decimals or to counts before you start.

Remember these

• $P(E_i \mid A) = \dfrac{P(E_i)\,P(A \mid E_i)}{\sum_j P(E_j)\,P(A \mid E_j)}$.
• Prior, likelihood and posterior are the three names to know.
• The posteriors of a partition add up to $1$.
• With natural frequencies, count the people on the paths that end in $A$, and then divide.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. A fair coin decides which box to use. Box $1$ holds $4$ red and $6$ blue balls. Box $2$ holds $7$ red and $3$ blue balls. A red ball is drawn. Find the probability that it came from box $1$.

2. Factory A makes $60\%$ of the screws and factory B makes the rest. A fault rate of $1\%$ holds for A and $4\%$ for B. A screw is found faulty. Find the probability that it came from B.

3. One person in $500$ has a disease. A test finds $98\%$ of the ill and wrongly flags $4\%$ of the healthy. Out of $100\,000$ people, how many test positive, and what is the chance that a person with a positive test is ill?

4. A test result is equally likely whether or not a person is ill. After a positive result, what is the posterior probability of being ill?

5. Three urns are chosen with equal probability. They hold red balls in the shares $\dfrac{1}{5}$, $\dfrac{2}{5}$ and $\dfrac{3}{5}$. A red ball is drawn. Find the probability that it came from the third urn.

References

• OpenStax, Introductory Statistics, Chapter 3, section on contingency tables and tree diagrams, where reversed conditional probabilities are found.
• NCERT, Mathematics Part II for Class XII, Chapter 13, section on Bayes' theorem.