Total Probability and Bayes' Theorem: Quick Revision

A brief note on Total Probability and Bayes' Theorem. Read it before a test. Try each question at the end before you open its answer.

Total probability

• Partition: events $E_1, \dots, E_n$ that are mutually exclusive, exhaustive and have positive probabilities.
• Law: $P(A) = \sum_i P(E_i)\,P(A \mid E_i)$.
• Two cases: $P(A) = P(B)\,P(A \mid B) + P(B')\,P(A \mid B')$.
• Tree: multiply along each path and add every path that ends in $A$.

Bayes' theorem

• Formula: $P(E_i \mid A) = \dfrac{P(E_i)\,P(A \mid E_i)}{\sum_j P(E_j)\,P(A \mid E_j)}$.
• Names: prior $P(E_i)$, likelihood $P(A \mid E_i)$, posterior $P(E_i \mid A)$.
• Natural frequencies: take a large round number of cases, count the cases on each path to $A$, and divide.
• Sum: the posteriors of a partition add up to $1$.

Tests and traps

• Sensitivity: $P(+ \mid \text{ill})$. Specificity: $P(- \mid \text{healthy})$.
• Base rate: a positive result for a rare illness is often a false alarm.
• Prosecutor's fallacy: $P(\text{evidence} \mid \text{innocent})$ is not $P(\text{innocent} \mid \text{evidence})$.
• Three doors: a host who knows where the prize is makes switching win with probability $\dfrac{2}{3}$.
• Second test: the posterior of the first test is the prior of the second, for independent tests.

A worked example

Two bags are chosen by the toss of a fair coin. Bag $1$ holds $3$ red and $1$ blue ball. Bag $2$ holds $1$ red and $3$ blue balls. A ball is drawn and it is red.

1. The paths to red. Bag $1$: $\dfrac{1}{2} \times \dfrac{3}{4} = \dfrac{3}{8}$. Bag $2$: $\dfrac{1}{2} \times \dfrac{1}{4} = \dfrac{1}{8}$.
2. The total. $P(\text{red}) = \dfrac{3}{8} + \dfrac{1}{8} = \dfrac{1}{2}$.
3. The posterior. $P(\text{bag } 1 \mid \text{red}) = \dfrac{3/8}{1/2} = \dfrac{3}{4}$.

Practice with new numbers

A program to try

The program finds the three answers of the worked example with exact fractions.

Mistakes to avoid

• Turning the condition around. $P(+ \mid \text{ill})$ is not $P(\text{ill} \mid +)$.
• Dividing by one path. The bottom of Bayes' theorem is the sum of all the paths to $A$.
• Ignoring the prior. Rare causes stay unlikely.
• Cases that overlap or leave a gap. Check that the shares add up to $1$.

Check yourself

1. Given $P(B) = 0.4$, $P(A \mid B) = 0.5$ and $P(A \mid B') = 0.25$, find $P(A)$.

2. For the same events, find $P(B \mid A)$.

3. A disease affects $4\%$ of a group. A test finds $80\%$ of the ill people and clears $95\%$ of the healthy people. A person tests positive. What is the probability that the person is ill?

4. Why do the posteriors of a partition add up to $1$?

5. In a game with $5$ doors, you pick one. The host, who knows where the prize is, opens $3$ other doors, all empty. You switch to the last closed door. What is the probability that you win?

Sources

• OpenStax, Introductory Statistics, Chapter 3, sections on the rules of probability, contingency tables and tree diagrams.
• NCERT, Mathematics Part II for Class XII, Chapter 13, sections on the theorem of total probability and Bayes' theorem.